A talk given in the University of Leeds School of Computing, on the nature of extended search sessions, and on evaluating/measuring learning/sensemaking during longer research sessions.
This document outlines a lesson plan for a 10th grade English class in Indonesia. The lesson focuses on a news item text about a meteor exploding over Russia. Students will watch a video about the event, fill in a blank text based on an audio recording, and answer comprehension questions in groups. The goals are for students to complete the text, answer questions about the news item, and identify implied information.
Jonathan Eisen talk for #IDWeek: Facilitating the Spread of Scientific Knowle...Jonathan Eisen
This document discusses the speaker's experience and advocacy for open science. It describes his early involvement with the Public Library of Science in advocating for open access publishing. It details an experiment publishing the first Wolbachia genome openly that helped convince him of the benefits of open science. The speaker discusses barriers to open science like paywalls limiting access to research and the need to make literature, data, software and other materials more openly available to benefit science and education.
The document discusses replication in HCI research. It notes that replication is important for science but is not facilitated well. Barriers include proprietary data/tools and difficulty reproducing studies exactly. Replication could create conflict or doubt novelty claims. The CHI conference series addressed this issue with a panel in 2011 and aims to further the discussion with RepliCHI, which would invite extended abstracts about replication attempts and present them at a dedicated event. The goal is to promote open discussion and incentivize replication to improve the field.
A talk given to local Small to Medium Enterprises about use of social media in business. Sponsored and Organised by Software Alliance Wales - http://softwarealliancewales.co.uk/
Casual-Leisure Search - Enterprise Search London MeetupMax L. Wilson
1. The document discusses two studies that uncovered casual-leisure search behaviors that current exploratory search models do not capture well.
2. These behaviors include need-less browsing where the goal is passing time rather than finding specific information, as well as exploring for the experience of exploring new spaces with family and friends.
3. The author proposes a definition of "casual-leisure search" that could help improve models of exploratory search to better reflect these uncommon search scenarios uncovered by the studies.
Max L. Wilson of Swansea University presented on evaluating the cognitive impact of search user interface design decisions. He discussed measuring cognitive load using techniques like brain scanning to objectively measure the effects of design changes on users' cognitive load. His research aims to produce clear design recommendations for user interfaces based on objective measures of cognitive impact.
The document outlines a series of 5 sessions. Each session lists the speakers in order from 1 to 4. The document is signed off by Max L. Wilson and includes his email address. The structure and information is repeated for each session.
IIiX2012 - Information vs Interaction - Examining different interaction model...Max L. Wilson
A talk given at the 2012 Information Interaction in Context conference (IIiX2012) where we developed 3 alternative versions of Google with 3 different refinement interactions on the left. Each used the same metadata, and the study was designed to show that users can get achieve better performance with different interaction over the same metadata.
You can provide benefits for searchers just be adjusting the interaction to your metadata. You do not _require_ better metadata to get better interaction.
This document outlines a lesson plan for a 10th grade English class in Indonesia. The lesson focuses on a news item text about a meteor exploding over Russia. Students will watch a video about the event, fill in a blank text based on an audio recording, and answer comprehension questions in groups. The goals are for students to complete the text, answer questions about the news item, and identify implied information.
Jonathan Eisen talk for #IDWeek: Facilitating the Spread of Scientific Knowle...Jonathan Eisen
This document discusses the speaker's experience and advocacy for open science. It describes his early involvement with the Public Library of Science in advocating for open access publishing. It details an experiment publishing the first Wolbachia genome openly that helped convince him of the benefits of open science. The speaker discusses barriers to open science like paywalls limiting access to research and the need to make literature, data, software and other materials more openly available to benefit science and education.
The document discusses replication in HCI research. It notes that replication is important for science but is not facilitated well. Barriers include proprietary data/tools and difficulty reproducing studies exactly. Replication could create conflict or doubt novelty claims. The CHI conference series addressed this issue with a panel in 2011 and aims to further the discussion with RepliCHI, which would invite extended abstracts about replication attempts and present them at a dedicated event. The goal is to promote open discussion and incentivize replication to improve the field.
A talk given to local Small to Medium Enterprises about use of social media in business. Sponsored and Organised by Software Alliance Wales - http://softwarealliancewales.co.uk/
Casual-Leisure Search - Enterprise Search London MeetupMax L. Wilson
1. The document discusses two studies that uncovered casual-leisure search behaviors that current exploratory search models do not capture well.
2. These behaviors include need-less browsing where the goal is passing time rather than finding specific information, as well as exploring for the experience of exploring new spaces with family and friends.
3. The author proposes a definition of "casual-leisure search" that could help improve models of exploratory search to better reflect these uncommon search scenarios uncovered by the studies.
Max L. Wilson of Swansea University presented on evaluating the cognitive impact of search user interface design decisions. He discussed measuring cognitive load using techniques like brain scanning to objectively measure the effects of design changes on users' cognitive load. His research aims to produce clear design recommendations for user interfaces based on objective measures of cognitive impact.
The document outlines a series of 5 sessions. Each session lists the speakers in order from 1 to 4. The document is signed off by Max L. Wilson and includes his email address. The structure and information is repeated for each session.
IIiX2012 - Information vs Interaction - Examining different interaction model...Max L. Wilson
A talk given at the 2012 Information Interaction in Context conference (IIiX2012) where we developed 3 alternative versions of Google with 3 different refinement interactions on the left. Each used the same metadata, and the study was designed to show that users can get achieve better performance with different interaction over the same metadata.
You can provide benefits for searchers just be adjusting the interaction to your metadata. You do not _require_ better metadata to get better interaction.
Academic Book Review Format. Book Review ExamJulie Gonzalez
The document provides instructions for creating an account on HelpWriting.net to request academic writing assistance. It outlines a 5-step process: 1) Create an account with email and password; 2) Complete a form with assignment details and attach a sample if desired; 3) Review writer bids and choose one; 4) Review the completed paper and authorize payment; 5) Request revisions until satisfied. It emphasizes the site's commitment to original, high-quality work and full refunds for plagiarized content.
This document provides resources and guidance for library research for a COUN 617 course. It introduces the reference librarian contact, Florence Tang, and provides her contact information. It also defines the role of a reference librarian according to Cornell University. The document then provides information on the reference desk hours and contact information, as well as tips on starting research early and resources that may not be available online or at all hours, like interlibrary loan. It also lists some databases relevant to the topic, such as PsycINFO, and provides links and login information for other resources.
The document discusses different types of research including descriptive research, exploratory research, quantitative research, and qualitative research. Descriptive research involves collecting quantitative or qualitative data to describe categories of information. Exploratory research examines a subject to gain further insight and identify issues for future research without seeking definitive answers. Quantitative research relies on quantitative data and statistics while qualitative research relies on qualitative data like interviews and observations to understand concepts, beliefs, and experiences. Examples of different types of research studies are provided.
The document discusses evidence-based practice and the expectations for first-year higher education students. It explores how the information seeking habits of the "Google Generation" can present challenges to evidence-based practice. Specifically, students may have difficulty evaluating sources for quality and appropriateness due to overreliance on sources like Wikipedia and Google. The document emphasizes the need to teach students critical thinking and evaluation skills so they can assess what counts as valid evidence and support their academic work with appropriate sources.
The document summarizes a presentation given on using Guided Inquiry to facilitate student inquiry and provide data for evidence-based practice. It describes a 2008 project involving multiple schools that used Guided Inquiry and wikis to scaffold student research. It also details a 2010 project at Loreto Kirribilli involving student research on the Berlin Wall. Analysis of student reflections and work provided insights into how Guided Inquiry can be improved to better support students' information search process and the development of their research skills.
Pick College Essay Writing Services With Care - Research Master EssaysCarla Bennington
This document provides instructions for ordering writing assistance from HelpWriting.net. It outlines a 5-step process: 1) Create an account with a password and email. 2) Complete a 10-minute order form providing instructions, sources, and deadline. 3) Review bids from writers and choose one based on qualifications. 4) Receive the paper and authorize payment if pleased. 5) Request revisions until fully satisfied, with a refund option for plagiarized work. The service utilizes a bidding system and promises original, high-quality content.
This document provides information and examples about the Explore-Flip-Apply instructional model. It discusses starting with a hooking activity to introduce a topic, having students research and fill in gaps on their own, and applying their learning through activities like discussions, debates, or role-plays. An example is given of a "fishbowl" activity where Chinese emperors discuss governmental topics. The document also provides tips for effective Google searches and references additional educational resources and websites.
This document describes a project between Western Washington University and local community colleges to improve science education for future elementary school teachers. The project developed hands-on, inquiry-based science curricula focusing on addressing students' preconceptions, developing conceptual understanding, and promoting metacognition. Students taking the new curricula had significantly higher science content learning and test score gains compared to those in traditional lecture courses. The curricula are now required or recommended for all elementary education majors at WWU and connect to courses at partner community colleges.
Stat 1040, Recitation packet 11. A 1999 study claimed that.docxdessiechisomjj4
Stat 1040, Recitation packet 1
1. A 1999 study claimed that
Infants who sleep at night in a bedroom with a light on may be at higher risk for myopia (nearsight-
edness) later in childhood.
The researchers surveyed parents of 479 children aged 2 to 16 seen in the ophthalmology outpatient
department of a children’s hospital. A questionnaire asked about the child’s nighttime light exposure
at the time of the survey and before age two. They noticed a positive association between myopia
and nighttime light exposure.
(a) Explain how you know that this is an observational study.
(b) Explain why this is not strong evidence that sleeping with a light on causes myopia by suggesting
a possible confounding factor and explaining clearly how this confounding factor could account
for the association they observed.
2. The following paragraph appears on the website www.alternative-medicine-and-health.com
Elmer Cranton, M.D., in his book, “Bypassing Bypass”, indicates that a ten year, 24
million dollar study conducted by the National Heart, Lung and Blood Institute, which
screened 16,000 patients who underwent coronary artery bypass at eleven leading medical
centers, revealed no increase in post-surgical survival rates as compared with a matched
group of non-surgically treated patients.
You may assume that the “matched group” was selected to resemble the original 16,000 with respect
to age, sex and type of heart disease.
(a) Based on what you read in the paragraph, was the study randomized? Explain clearly.
(b) Was the study blind? Explain clearly.
(c) Explain the major problem with a study such as this one, and why it would probably not give
very reliable results.
3. A recent study in Europe looked at a large group of women of childbearing age. The researchers asked
each woman how much alcohol they had consumed over the past 12 months. The researchers found
that women who drank moderate amounts of alcohol were somewhat less likely to have infertility than
women who did not (November, 2001). The study said it “controlled for age, income and religion”.
(a) Based on the information above, was this a controlled experiment or an observational study?
(b) Why did they “control for” age, income and religion?
(c) Is this convincing evidence that infertility would decrease if women with infertility started to
drink moderate amounts of alcohol? (Note: we are only asking about infertility. There may be
other problems introduced by such behavior, but ignore these for answering this question).
(d) Suggest a possible confounding factor (other than age, income, or religion) and clearly explain
why you think it might be a confounding factor.
4. A randomized, controlled, double-blind study published in March, 2008 shows the well-known “placebo
effect” works even better if the placebo costs more. In the study, volunteers were given an electric
shock and took a pill. Volunteers in the treatment group were told it was an expensive painkiller,
while those in the c.
This document appears to be notes from a presentation or workshop on teaching methods. It discusses the Explore-Flip-Apply instructional model, providing examples of how to structure lessons around an initial exploration activity, having students research content on their own, and then applying what they've learned in class. Other topics mentioned include Bloom's Taxonomy, cognitive load, and Google search skills. Several education experts and resources are also listed for further information.
This document appears to be notes from a presentation or workshop on teaching methods. It discusses the Explore-Flip-Apply instructional model, providing examples of how to structure lessons around an initial exploration activity, additional preparation done outside of class ("flipping"), and application activities done in class. Other topics mentioned include Bloom's Taxonomy, cognitive load, and search skills for using Google effectively in instruction. Helpful education resources and websites are also listed.
Illinois Mathematics & Science Academy (IMSA) graduate Ziang Wang is a member of The National Society of High School Scholars and a 2015 recipient of the NSHSS Foundation's annual Earth Day Award. In this Q&A, Ziang discusses his project on green roofs and pollution problems in China.
Dr. Sohail Tavazoie is a successful cancer researcher who has made significant contributions to understanding how small RNAs regulate gene expression and cancer metastasis. In an interview, he discusses his passion for science from an early age, his impressive career accomplishments, and his continued motivation to develop more effective cancer therapies for patients. He provides advice for young researchers to find questions they are passionate about, communicate their work effectively, and develop resilience when facing rejections.
Experiments: The Good, the Bad, and the BeautifulTechWell
Through the years, Linda Rising has given presentations about the use of stories instead of science in the industry, so in this session she has decided to be more helpful and talk about experiments. There's an increasing emphasis on experiments as a part of being more innovative but sometimes Linda says we need a nudge and some examples to help us get going. No, this is not too rigorous! Rather than talking about statistics, she is going to explore cheap, easy experiments—what to do, what to be aware of, and our own cognitive biases, including the confirmation bias that does its best to keep us from seeing what's new in our environment. We all need strategies for dealing with that—like involving others who are really doing it. Linda’s goal is to encourage everyone to be a bit more methodical in decision-making and to replace “That won't work” with “How can we test it?” Leave with a plan for one or more experiments to run in your workplace. Improve your scientific vocabulary a bit, and learn some of the cognitive biases that get in the way of good decision-making.
This document discusses factors that influence intellectual formation and education outcomes. It summarizes data showing that students in top-performing countries spend more time in school each year and do less homework per week compared to US students. It also notes that other countries have higher ratios of computers to students. The document suggests family breakdown is a contributing factor to challenges in intellectual formation, as stress from familial discord impacts brain development and cognitive functioning. Simply increasing school hours and homework may not address underlying issues and could increase stress levels, rather than improving outcomes. A holistic and nuanced approach is needed.
This is a keynote presentation to "Open science, transparence et evaluation. Perspectives et enjeux pour les chercheurs.' Held at Urfist de Bordeaux, France, 4 April 2017
https://sygefor.reseau-urfist.fr/#!/training/6701/7159/?from=true
ABSTRACT: The way research is disseminated has changed immeasurably since the advent of the internet, yet we still reward researchers in the same way - for publication of novel results in high impact journals. This talk will start with a brief discussion of some of the big challenges the research sector is facing as a result and describe how Open Science can address these. The talk will then focus on the difficulty of introducing and implementing Open Science solutions. Open Science questions the status quo, and potentially threatens the established reputation of both institutions and individuals. It is not an easy concept to implement. While the discipline of Scholarly Communication takes a 'meta' view of the whole research ecosystem, most players in that system are working within a narrow view. It is very rare for individuals to be able to see beyond their own experience. Challenges for people trying to implement Open Science initiatives range from practical issues in implementing change, through to the people skills and negotiations required to convince individuals and institutions that this change is necessary.
Philip Bourne, Ph.D.
University of California, San Diego
The Future of Data Sharing Platforms in Academic Medicine: A Basic Science Perspective
Meeting: Physician-Scientist Career Development Meeting, New York Academy of Sciences, November 3-5, 2010
View this online, complete with audio, at http://community.sciencecareers.org/ctscinet/groups/sessions/2010/12/the-future-of-data-sharing-platforms-in-academic-medicine-a-basic-science-perspective.php
Brain Data as Cognitive Personal Informatics - UCL 2022Max L. Wilson
A human-computer interaction research talk about how we measure mental workload, and how people might reflect on this type of personal data in the future. The research is carried out at the University of Nottingham in the School of Computer Science. Based on a study published at CHI2022, led by Serena Midha.
Brain Data as Cognitive Personal Informatics - Bell Labs 2022Max L. Wilson
A human-computer interaction research talk about how we measure mental workload, and how people might reflect on this type of personal data in the future. The research is carried out at the University of Nottingham in the School of Computer Science. Based on a study published at CHI2022, led by Serena Midha. Presented to Bell Labs
More Related Content
Similar to Understanding & Evaluating Search Sessions
Academic Book Review Format. Book Review ExamJulie Gonzalez
The document provides instructions for creating an account on HelpWriting.net to request academic writing assistance. It outlines a 5-step process: 1) Create an account with email and password; 2) Complete a form with assignment details and attach a sample if desired; 3) Review writer bids and choose one; 4) Review the completed paper and authorize payment; 5) Request revisions until satisfied. It emphasizes the site's commitment to original, high-quality work and full refunds for plagiarized content.
This document provides resources and guidance for library research for a COUN 617 course. It introduces the reference librarian contact, Florence Tang, and provides her contact information. It also defines the role of a reference librarian according to Cornell University. The document then provides information on the reference desk hours and contact information, as well as tips on starting research early and resources that may not be available online or at all hours, like interlibrary loan. It also lists some databases relevant to the topic, such as PsycINFO, and provides links and login information for other resources.
The document discusses different types of research including descriptive research, exploratory research, quantitative research, and qualitative research. Descriptive research involves collecting quantitative or qualitative data to describe categories of information. Exploratory research examines a subject to gain further insight and identify issues for future research without seeking definitive answers. Quantitative research relies on quantitative data and statistics while qualitative research relies on qualitative data like interviews and observations to understand concepts, beliefs, and experiences. Examples of different types of research studies are provided.
The document discusses evidence-based practice and the expectations for first-year higher education students. It explores how the information seeking habits of the "Google Generation" can present challenges to evidence-based practice. Specifically, students may have difficulty evaluating sources for quality and appropriateness due to overreliance on sources like Wikipedia and Google. The document emphasizes the need to teach students critical thinking and evaluation skills so they can assess what counts as valid evidence and support their academic work with appropriate sources.
The document summarizes a presentation given on using Guided Inquiry to facilitate student inquiry and provide data for evidence-based practice. It describes a 2008 project involving multiple schools that used Guided Inquiry and wikis to scaffold student research. It also details a 2010 project at Loreto Kirribilli involving student research on the Berlin Wall. Analysis of student reflections and work provided insights into how Guided Inquiry can be improved to better support students' information search process and the development of their research skills.
Pick College Essay Writing Services With Care - Research Master EssaysCarla Bennington
This document provides instructions for ordering writing assistance from HelpWriting.net. It outlines a 5-step process: 1) Create an account with a password and email. 2) Complete a 10-minute order form providing instructions, sources, and deadline. 3) Review bids from writers and choose one based on qualifications. 4) Receive the paper and authorize payment if pleased. 5) Request revisions until fully satisfied, with a refund option for plagiarized work. The service utilizes a bidding system and promises original, high-quality content.
This document provides information and examples about the Explore-Flip-Apply instructional model. It discusses starting with a hooking activity to introduce a topic, having students research and fill in gaps on their own, and applying their learning through activities like discussions, debates, or role-plays. An example is given of a "fishbowl" activity where Chinese emperors discuss governmental topics. The document also provides tips for effective Google searches and references additional educational resources and websites.
This document describes a project between Western Washington University and local community colleges to improve science education for future elementary school teachers. The project developed hands-on, inquiry-based science curricula focusing on addressing students' preconceptions, developing conceptual understanding, and promoting metacognition. Students taking the new curricula had significantly higher science content learning and test score gains compared to those in traditional lecture courses. The curricula are now required or recommended for all elementary education majors at WWU and connect to courses at partner community colleges.
Stat 1040, Recitation packet 11. A 1999 study claimed that.docxdessiechisomjj4
Stat 1040, Recitation packet 1
1. A 1999 study claimed that
Infants who sleep at night in a bedroom with a light on may be at higher risk for myopia (nearsight-
edness) later in childhood.
The researchers surveyed parents of 479 children aged 2 to 16 seen in the ophthalmology outpatient
department of a children’s hospital. A questionnaire asked about the child’s nighttime light exposure
at the time of the survey and before age two. They noticed a positive association between myopia
and nighttime light exposure.
(a) Explain how you know that this is an observational study.
(b) Explain why this is not strong evidence that sleeping with a light on causes myopia by suggesting
a possible confounding factor and explaining clearly how this confounding factor could account
for the association they observed.
2. The following paragraph appears on the website www.alternative-medicine-and-health.com
Elmer Cranton, M.D., in his book, “Bypassing Bypass”, indicates that a ten year, 24
million dollar study conducted by the National Heart, Lung and Blood Institute, which
screened 16,000 patients who underwent coronary artery bypass at eleven leading medical
centers, revealed no increase in post-surgical survival rates as compared with a matched
group of non-surgically treated patients.
You may assume that the “matched group” was selected to resemble the original 16,000 with respect
to age, sex and type of heart disease.
(a) Based on what you read in the paragraph, was the study randomized? Explain clearly.
(b) Was the study blind? Explain clearly.
(c) Explain the major problem with a study such as this one, and why it would probably not give
very reliable results.
3. A recent study in Europe looked at a large group of women of childbearing age. The researchers asked
each woman how much alcohol they had consumed over the past 12 months. The researchers found
that women who drank moderate amounts of alcohol were somewhat less likely to have infertility than
women who did not (November, 2001). The study said it “controlled for age, income and religion”.
(a) Based on the information above, was this a controlled experiment or an observational study?
(b) Why did they “control for” age, income and religion?
(c) Is this convincing evidence that infertility would decrease if women with infertility started to
drink moderate amounts of alcohol? (Note: we are only asking about infertility. There may be
other problems introduced by such behavior, but ignore these for answering this question).
(d) Suggest a possible confounding factor (other than age, income, or religion) and clearly explain
why you think it might be a confounding factor.
4. A randomized, controlled, double-blind study published in March, 2008 shows the well-known “placebo
effect” works even better if the placebo costs more. In the study, volunteers were given an electric
shock and took a pill. Volunteers in the treatment group were told it was an expensive painkiller,
while those in the c.
This document appears to be notes from a presentation or workshop on teaching methods. It discusses the Explore-Flip-Apply instructional model, providing examples of how to structure lessons around an initial exploration activity, having students research content on their own, and then applying what they've learned in class. Other topics mentioned include Bloom's Taxonomy, cognitive load, and Google search skills. Several education experts and resources are also listed for further information.
This document appears to be notes from a presentation or workshop on teaching methods. It discusses the Explore-Flip-Apply instructional model, providing examples of how to structure lessons around an initial exploration activity, additional preparation done outside of class ("flipping"), and application activities done in class. Other topics mentioned include Bloom's Taxonomy, cognitive load, and search skills for using Google effectively in instruction. Helpful education resources and websites are also listed.
Illinois Mathematics & Science Academy (IMSA) graduate Ziang Wang is a member of The National Society of High School Scholars and a 2015 recipient of the NSHSS Foundation's annual Earth Day Award. In this Q&A, Ziang discusses his project on green roofs and pollution problems in China.
Dr. Sohail Tavazoie is a successful cancer researcher who has made significant contributions to understanding how small RNAs regulate gene expression and cancer metastasis. In an interview, he discusses his passion for science from an early age, his impressive career accomplishments, and his continued motivation to develop more effective cancer therapies for patients. He provides advice for young researchers to find questions they are passionate about, communicate their work effectively, and develop resilience when facing rejections.
Experiments: The Good, the Bad, and the BeautifulTechWell
Through the years, Linda Rising has given presentations about the use of stories instead of science in the industry, so in this session she has decided to be more helpful and talk about experiments. There's an increasing emphasis on experiments as a part of being more innovative but sometimes Linda says we need a nudge and some examples to help us get going. No, this is not too rigorous! Rather than talking about statistics, she is going to explore cheap, easy experiments—what to do, what to be aware of, and our own cognitive biases, including the confirmation bias that does its best to keep us from seeing what's new in our environment. We all need strategies for dealing with that—like involving others who are really doing it. Linda’s goal is to encourage everyone to be a bit more methodical in decision-making and to replace “That won't work” with “How can we test it?” Leave with a plan for one or more experiments to run in your workplace. Improve your scientific vocabulary a bit, and learn some of the cognitive biases that get in the way of good decision-making.
This document discusses factors that influence intellectual formation and education outcomes. It summarizes data showing that students in top-performing countries spend more time in school each year and do less homework per week compared to US students. It also notes that other countries have higher ratios of computers to students. The document suggests family breakdown is a contributing factor to challenges in intellectual formation, as stress from familial discord impacts brain development and cognitive functioning. Simply increasing school hours and homework may not address underlying issues and could increase stress levels, rather than improving outcomes. A holistic and nuanced approach is needed.
This is a keynote presentation to "Open science, transparence et evaluation. Perspectives et enjeux pour les chercheurs.' Held at Urfist de Bordeaux, France, 4 April 2017
https://sygefor.reseau-urfist.fr/#!/training/6701/7159/?from=true
ABSTRACT: The way research is disseminated has changed immeasurably since the advent of the internet, yet we still reward researchers in the same way - for publication of novel results in high impact journals. This talk will start with a brief discussion of some of the big challenges the research sector is facing as a result and describe how Open Science can address these. The talk will then focus on the difficulty of introducing and implementing Open Science solutions. Open Science questions the status quo, and potentially threatens the established reputation of both institutions and individuals. It is not an easy concept to implement. While the discipline of Scholarly Communication takes a 'meta' view of the whole research ecosystem, most players in that system are working within a narrow view. It is very rare for individuals to be able to see beyond their own experience. Challenges for people trying to implement Open Science initiatives range from practical issues in implementing change, through to the people skills and negotiations required to convince individuals and institutions that this change is necessary.
Philip Bourne, Ph.D.
University of California, San Diego
The Future of Data Sharing Platforms in Academic Medicine: A Basic Science Perspective
Meeting: Physician-Scientist Career Development Meeting, New York Academy of Sciences, November 3-5, 2010
View this online, complete with audio, at http://community.sciencecareers.org/ctscinet/groups/sessions/2010/12/the-future-of-data-sharing-platforms-in-academic-medicine-a-basic-science-perspective.php
Similar to Understanding & Evaluating Search Sessions (20)
Brain Data as Cognitive Personal Informatics - UCL 2022Max L. Wilson
A human-computer interaction research talk about how we measure mental workload, and how people might reflect on this type of personal data in the future. The research is carried out at the University of Nottingham in the School of Computer Science. Based on a study published at CHI2022, led by Serena Midha.
Brain Data as Cognitive Personal Informatics - Bell Labs 2022Max L. Wilson
A human-computer interaction research talk about how we measure mental workload, and how people might reflect on this type of personal data in the future. The research is carried out at the University of Nottingham in the School of Computer Science. Based on a study published at CHI2022, led by Serena Midha. Presented to Bell Labs
Physiological indicators of task demand, fatigue, and cognition during Work T...Max L. Wilson
A human-computer interaction research talk about how we measure mental workload, and how people might reflect on this type of personal data in the future. The research is carried out at the University of Nottingham in the School of Computer Science, involving functional Near Infrared Spectroscopy (fNIRS)
Brain-based HCI - What brain data can tell us about HCI - St Andrews, 2019Max L. Wilson
A human-computer interaction research talk about how we measure mental workload, and how people might reflect on this type of personal data in the future. The research is carried out at the University of Nottingham in the School of Computer Science, involving functional Near Infrared Spectroscopy (fNIRS)
Mental Workload Alerts - Reliable Brain Measurements of HCI using fNIRS - Lei...Max L. Wilson
A human-computer interaction research talk about how we measure mental workload, and how people might reflect on this type of personal data in the future. The research is carried out at the University of Nottingham in the School of Computer Science, involving functional Near Infrared Spectroscopy (fNIRS)
Mental Workload Alerts - Reliable Brain Measurements of HCI using fNIRS - Uni...Max L. Wilson
A human-computer interaction research talk about how we measure mental workload, and how people might reflect on this type of personal data in the future. The research is carried out at the University of Nottingham in the School of Computer Science, involving functional Near Infrared Spectroscopy (fNIRS)
Measuring & Reflecting on Mental Workload - Birmingham Uni, May 2017Max L. Wilson
A human-computer interaction research talk about how we measure mental workload, and how people might reflect on this type of personal data in the future. The research is carried out at the University of Nottingham in the School of Computer Science, involving functional Near Infrared Spectroscopy (fNIRS)
CHIIR2017 - Tetris Model of Resolving Information NeedsMax L. Wilson
The document introduces the "Tetris Model" of resolving information needs. It uses the game of Tetris as a metaphor to show how information needs can vary in complexity, from simple lookups to more involved investigative searches. In the Tetris Model, simple information needs correspond to getting the right piece quickly to clear lines, while complex needs are like having to investigate different pieces to solve a more challenging part of the board. The model helps conceptualize how information needs can increase in depth and complexity over time as knowledge grows. However, it also has limitations and does not capture all aspects of information searching such as stages, actions, or time pressure.
The HCI Perspective on IR (DIR2016 Keynote)Max L. Wilson
The Human-Computer Interaction perspective on Information Retrieval, Keynote Speech presented at the 15th Dutch-Belgian Information Retrieval Workshop - Nov 25th 2016
RepliCHI - 8 Challenges in Replicating a StudyMax L. Wilson
This document summarizes Dr. Max Wilson's experience replicating a 2011 study by Shah and González-Ibáñez on collaborative information seeking using the Coagmento software. Some key challenges included using an updated version of the software, designing tasks that were still relevant, limited resources and timescales, and uncertainty around data processing and analysis methods. While the replication found different results, it provided valuable lessons about issues in replicating studies for both research and teaching purposes.
This document profiles Dr. Max L. Wilson, who researches search user interface design at the University of Nottingham. It provides an overview of his background and qualifications, research areas related to social media search and casual search, and frameworks for evaluating advanced search interfaces. The document also shares details about Dr. Wilson's publications and presentations, and invites the reader to learn more about his work through a video and website.
ASIST2010 - The Revisit Rack - Group Web Search ThumbnailsMax L. Wilson
A conference presentation at ASIST2010 about the (bad) idea of grouping web search thumbnails together at the top of Search Engine Results Pages. Just don't do it!
Investigating Alternative Forms of SearchMax L. Wilson
The document discusses the results of a study on the impact of COVID-19 lockdowns on air pollution. Researchers analyzed data from dozens of countries and found that lockdowns led to an average decline of nearly 30% in nitrogen dioxide levels across major cities. However, they also observed that the reductions in air pollution were temporary and that levels began to rise again as restrictions eased and human activity increased.
20 Comprehensive Checklist of Designing and Developing a WebsitePixlogix Infotech
Dive into the world of Website Designing and Developing with Pixlogix! Looking to create a stunning online presence? Look no further! Our comprehensive checklist covers everything you need to know to craft a website that stands out. From user-friendly design to seamless functionality, we've got you covered. Don't miss out on this invaluable resource! Check out our checklist now at Pixlogix and start your journey towards a captivating online presence today.
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slackshyamraj55
Discover the seamless integration of RPA (Robotic Process Automation), COMPOSER, and APM with AWS IDP enhanced with Slack notifications. Explore how these technologies converge to streamline workflows, optimize performance, and ensure secure access, all while leveraging the power of AWS IDP and real-time communication via Slack notifications.
In his public lecture, Christian Timmerer provides insights into the fascinating history of video streaming, starting from its humble beginnings before YouTube to the groundbreaking technologies that now dominate platforms like Netflix and ORF ON. Timmerer also presents provocative contributions of his own that have significantly influenced the industry. He concludes by looking at future challenges and invites the audience to join in a discussion.
Maruthi Prithivirajan, Head of ASEAN & IN Solution Architecture, Neo4j
Get an inside look at the latest Neo4j innovations that enable relationship-driven intelligence at scale. Learn more about the newest cloud integrations and product enhancements that make Neo4j an essential choice for developers building apps with interconnected data and generative AI.
UiPath Test Automation using UiPath Test Suite series, part 6DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 6. In this session, we will cover Test Automation with generative AI and Open AI.
UiPath Test Automation with generative AI and Open AI webinar offers an in-depth exploration of leveraging cutting-edge technologies for test automation within the UiPath platform. Attendees will delve into the integration of generative AI, a test automation solution, with Open AI advanced natural language processing capabilities.
Throughout the session, participants will discover how this synergy empowers testers to automate repetitive tasks, enhance testing accuracy, and expedite the software testing life cycle. Topics covered include the seamless integration process, practical use cases, and the benefits of harnessing AI-driven automation for UiPath testing initiatives. By attending this webinar, testers, and automation professionals can gain valuable insights into harnessing the power of AI to optimize their test automation workflows within the UiPath ecosystem, ultimately driving efficiency and quality in software development processes.
What will you get from this session?
1. Insights into integrating generative AI.
2. Understanding how this integration enhances test automation within the UiPath platform
3. Practical demonstrations
4. Exploration of real-world use cases illustrating the benefits of AI-driven test automation for UiPath
Topics covered:
What is generative AI
Test Automation with generative AI and Open AI.
UiPath integration with generative AI
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
UiPath Test Automation using UiPath Test Suite series, part 5DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 5. In this session, we will cover CI/CD with devops.
Topics covered:
CI/CD with in UiPath
End-to-end overview of CI/CD pipeline with Azure devops
Speaker:
Lyndsey Byblow, Test Suite Sales Engineer @ UiPath, Inc.
A tale of scale & speed: How the US Navy is enabling software delivery from l...sonjaschweigert1
Rapid and secure feature delivery is a goal across every application team and every branch of the DoD. The Navy’s DevSecOps platform, Party Barge, has achieved:
- Reduction in onboarding time from 5 weeks to 1 day
- Improved developer experience and productivity through actionable findings and reduction of false positives
- Maintenance of superior security standards and inherent policy enforcement with Authorization to Operate (ATO)
Development teams can ship efficiently and ensure applications are cyber ready for Navy Authorizing Officials (AOs). In this webinar, Sigma Defense and Anchore will give attendees a look behind the scenes and demo secure pipeline automation and security artifacts that speed up application ATO and time to production.
We will cover:
- How to remove silos in DevSecOps
- How to build efficient development pipeline roles and component templates
- How to deliver security artifacts that matter for ATO’s (SBOMs, vulnerability reports, and policy evidence)
- How to streamline operations with automated policy checks on container images
Pushing the limits of ePRTC: 100ns holdover for 100 daysAdtran
At WSTS 2024, Alon Stern explored the topic of parametric holdover and explained how recent research findings can be implemented in real-world PNT networks to achieve 100 nanoseconds of accuracy for up to 100 days.
Introducing Milvus Lite: Easy-to-Install, Easy-to-Use vector database for you...Zilliz
Join us to introduce Milvus Lite, a vector database that can run on notebooks and laptops, share the same API with Milvus, and integrate with every popular GenAI framework. This webinar is perfect for developers seeking easy-to-use, well-integrated vector databases for their GenAI apps.
GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...Neo4j
Leonard Jayamohan, Partner & Generative AI Lead, Deloitte
This keynote will reveal how Deloitte leverages Neo4j’s graph power for groundbreaking digital twin solutions, achieving a staggering 100x performance boost. Discover the essential role knowledge graphs play in successful generative AI implementations. Plus, get an exclusive look at an innovative Neo4j + Generative AI solution Deloitte is developing in-house.
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024Albert Hoitingh
In this session I delve into the encryption technology used in Microsoft 365 and Microsoft Purview. Including the concepts of Customer Key and Double Key Encryption.
For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2024/06/building-and-scaling-ai-applications-with-the-nx-ai-manager-a-presentation-from-network-optix/
Robin van Emden, Senior Director of Data Science at Network Optix, presents the “Building and Scaling AI Applications with the Nx AI Manager,” tutorial at the May 2024 Embedded Vision Summit.
In this presentation, van Emden covers the basics of scaling edge AI solutions using the Nx tool kit. He emphasizes the process of developing AI models and deploying them globally. He also showcases the conversion of AI models and the creation of effective edge AI pipelines, with a focus on pre-processing, model conversion, selecting the appropriate inference engine for the target hardware and post-processing.
van Emden shows how Nx can simplify the developer’s life and facilitate a rapid transition from concept to production-ready applications.He provides valuable insights into developing scalable and efficient edge AI solutions, with a strong focus on practical implementation.
“An Outlook of the Ongoing and Future Relationship between Blockchain Technologies and Process-aware Information Systems.” Invited talk at the joint workshop on Blockchain for Information Systems (BC4IS) and Blockchain for Trusted Data Sharing (B4TDS), co-located with with the 36th International Conference on Advanced Information Systems Engineering (CAiSE), 3 June 2024, Limassol, Cyprus.
1. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
Extended Searching Sessions
and Evaluating Success
Dr Max L.Wilson
Mixed Reality Lab
University of Nottingham, UK
Friday, 10 May 13
2. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
Studying Extended Search
Success In Observable Natural
Sessions
SESSIONS
Friday, 10 May 13
3. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
Extended Searching Sessions and
Evaluating Sensemaking Success
About Me
Study 1:The Real
Nature of Sessions
Study 2: Evaluating
Sensemaking Success
Friday, 10 May 13
4. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
About me
MEng & Phd in Southampton
Taught in Swansea for 3 years
Moved to Nottingham April 2012
Friday, 10 May 13
5. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
About Me
Friday, 10 May 13
7. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
UIST
2008
JCDL
2008
Friday, 10 May 13
8. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
My PhD
Bates, M. J. (1979a). Idea tactics. Journal of
the American Society for Information
Science, 30(5):280–289.
Bates, M. J. (1979b). Information search
tactics. Journal of the American Society for
Information Science, 30(4):205–214.
Belkin, N. J., Marchetti, P. G., and Cool, C.
(1993). Braque: design of an interface to support
user interaction in information retrieval.
Information Processing and Management, 29(3):
325–344.
Friday, 10 May 13
9. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
My PhD
Wilson, M. L., schraefel, m. c., and White, R. W. (2009). Evaluating advanced
search interfaces using established information-seeking models. Journal of the
American Society for Information Science and Technology, 60(7):1407–1422.
Friday, 10 May 13
10. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
Search User Interface Design
Friday, 10 May 13
11. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
MyTeam
Horia Maior Matthew Pike Jon Hurlock Paul BrindleyZenah Alkubaisy
Chaoyu (Kelvin)Ye
(Study 1)
Mathew Wilson
(Study 2)
Friday, 10 May 13
12. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
Extended Searching Sessions and
Evaluating Sensemaking Success
About Me
Study 1:The Real
Nature of Sessions
Study 2: Evaluating
Sensemaking Success
Friday, 10 May 13
13. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
People Searching the Web
Elsweiler, D.,Wilson M. L. and Kirkegaard-Lunn, B. (2011) Understanding Casual-leisure Information Behaviour. In Spink,A.
and Heinstrom, J. (Eds) New Directions in Information Behaviour. Emerald Group Publishing Limited, pp 211-241.
Friday, 10 May 13
14. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
The Search Communities
Ingwersen, P., Jarvelin, K., 2005.The turn:
integration of information seeking
and retrieval in context. Springer, Berlin, Germany.
The IR Community
•Focused on Accuracy
•Are these results relevant?
•How many are relevant?
•Did we get all the relevant ones?
Friday, 10 May 13
15. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
The Search Communities
The IS Community
•Focused on Success
•Did they find the right result?
•How long did they take
•How many interactions?
Ingwersen, P., Jarvelin, K., 2005.The turn:
integration of information seeking
and retrieval in context. Springer, Berlin, Germany.
Friday, 10 May 13
16. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
The Search Communities
The IB Community
•Focused on Quality
•Did they do a good job?
•How did the UI affect the task?
•Was the higher level motivating
task achieved more successfully?
Ingwersen, P., Jarvelin, K., 2005.The turn:
integration of information seeking
and retrieval in context. Springer, Berlin, Germany.
Friday, 10 May 13
17. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
The Search Communities
“Relatively” well known
“Naively estimated”
- Study 1
“Simplistically” measured
- Study 2
Friday, 10 May 13
18. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
WorkTasks
• Work tasks - typically considered work-led information-
intensive activities the lead to searching
• Can be out-of-work - like planning holidays, or buying a car
• We’ve begun looking at motivating ‘tasks’ outside of work
Friday, 10 May 13
19. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
Casual Leisure WorkTasks
behaviours documented so far.
4.1 Need-less browsing
Much like the desire to pass time at the television, we saw
many examples (some shown in Table 3) of people passing
time typically associated with the ‘browsing’ keyword.
1) ... I’m not even *doing* anything useful... just browsing
eBay aimlessly...
2) to do list today: browse the Internet until fasting break
time..
3) ... just got done eating dinner and my family is watch-
ing the football. Rather browse on the laptop
4) I’m at the dolphin mall. Just browsing.
Table 3: Example tweets where the browsing activ-
ity is need-less.
From the collected tweets it is clear that often the inform-
ation-need in these situations are not only fuzzy, but typi-
cally absent. The aim appears to be focused on the activity,
where the measure of success would be in how much they
D
d
a
5
h
f
o
S
i
b
a
d
f
t
s
W
t
Wilson, M. L. and Elsweiler, D. (2010) Casual-leisure Searching: the Exploratory Search scenarios
that break our current models. In: 4th HCIR Workshop ,Aug 22 2010. pp 28-31.
Friday, 10 May 13
20. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
People Searching the Web
Elsweiler, D.,Wilson M. L. and Kirkegaard-Lunn, B. (2011) Understanding Casual-leisure Information Behaviour. In Spink,A.
and Heinstrom, J. (Eds) New Directions in Information Behaviour. Emerald Group Publishing Limited, pp 211-241.
Friday, 10 May 13
21. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
• Traditionally examined by analysing logs for stats
• In the 90s, suggested they are broken by ~25mins
- More recently by ~5mins
• BUT evidence shows web use typically interleaves tasks
- AND tabs make this all much harder
• Become a big focus as Dagstuhls/workshops
Sessions
Friday, 10 May 13
22. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
SearchTrails
• Aimed at finding common
end locations for queries
• An interesting step towards
sessions though
• most involved some trail
features (not query+click)
White, Ryen W., and Steven M. Drucker. "Investigating behavioral variability in
web search." in Proc WWW 2007 .ACM
Friday, 10 May 13
23. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
Top Sessions
as Seen by Bing
Bailey et al, User task understanding: a web search engine perspective, NII Shonan, 8 Oct 2012
Friday, 10 May 13
24. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
Top Sessions
as Seen by Bing
Bailey et al, User task understanding: a web search engine perspective, NII Shonan, 8 Oct 2012
Friday, 10 May 13
25. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
Top Sessions
as Seen by Bing
Bailey et al, User task understanding: a web search engine perspective, NII Shonan, 8 Oct 2012
Friday, 10 May 13
26. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
Study 1:
Investigating Extended Sessions
What on earth
is happening here?
Friday, 10 May 13
27. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
Study 1: Interview Method
Send & Preprocess History
Interview Recording, Cards, Card Sorts, Marked history file, log data
A history artefact - approx 300 items
How
would
you
define a
session?
Mark out history into
sessions, starting recently
+
Create ‘cards’ of varying
types of ‘sessions’
Open Card Sort
+
Close Card Sort
10mins 20-30mins 30-50mins
15-20Cards
Friday, 10 May 13
28. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
Study 1: Data
• Rich discussion of ~20 Sessions per participant
• Currently: 7 participants and ~120 sessions
- richly described and compared
• Aiming for: 12 participants and 200+ sessions at first
Friday, 10 May 13
29. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
Study 1: Questions for Sessions
1)Where was this done (e.g. work vs home vs mobile)
2)With who (collaborative?)
3)For who (shared task?)
4)Devices involved (whether devices affect things)
5)Length of the Session (how do they define long?)
6)Successful or not (for future measurement insights)
At some point: tried to learn these for each session
Friday, 10 May 13
30. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
Study 1:A Card
Friday, 10 May 13
31. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
Study 1:A Card
Friday, 10 May 13
32. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
Study 1: Card Sorting
• We aimed first to let them define the dimensions
- this lets us see how they define things
- how do they self-categorise different sessions
• We then had some targeted card sorts
- For who, duration, difficulty, importance, location
- whats short vs long?
- whats important vs not?
- how do people divide work vs home etc
Friday, 10 May 13
33. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
Study 1: Example Card Sorts
Friday, 10 May 13
34. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
Friday, 10 May 13
35. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
Study 1: Preliminary Findings
• avg 21 cards per person, inc ~8 sessions of 5+mins
- ~4 work & ~4 leisure
• 18.6% of those extended sessions involved task switches
• avg length: 17.5mins avg #queries: 3.55
• short: third said <30s, third said <1m third said <30m
• long: third said >1hour, third said >5mins
Friday, 10 May 13
36. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
Study 1: Preliminary Findings
• longest sessions: entertainment, work prep, news, shopping
• longest leisure: 22-76mins youtube, 28mins news
• most important: work, money, urgent shopping
• lest important: leisure, entertainment, free time
• most difficult: technical work prep
Friday, 10 May 13
37. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
Study 1: Preliminary Findings
• Huge divide over where sessions start or stop
- many people considered a session to span a large break
- paused and left in tabs
• One person divided a single topical episode by phases
- and phases were sessions
- e.g. broadening/confused stage vs successful focus stage
• One person divided single topical episode by major sources
- moved from web searching to video searching on same topic
What is a session?
Implications for where/when to measure success
Friday, 10 May 13
38. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
Study 1:What is a session?
Single topic - changing purpose
Friday, 10 May 13
39. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
Study 1:What is a session?
Single topic - pausing sessions
Friday, 10 May 13
40. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
Study 1:What is a session?
Low-query extended sessions
Friday, 10 May 13
41. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
Study 1: Other observations
• Seeing an informal relationship between who tasks are for
- and skewed importance
- including for another person, or for a group
- and slow sequential interactions (as talk to others)
• Seeing a strong low-query correlation with entertainment
- seeing serious-leisure more similar to work tasks
• Hard tasks have high query loads,
- and are related to rare or new areas
Friday, 10 May 13
42. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
Study 1: Summary
• We’re beginning to get some real insight into real sessions
• Already identifying examples where time-splitting isnt sufficient
- but intention changing is common
• We’re seeing possible common patterns of overlapping sessions
• We havent finished!
Friday, 10 May 13
43. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
Study 2: Evaluating Sensemaking
“Simplistically” measured
- Study 2
Wilson, M. J. and Wilson, M. L. (2012) A Comparison ofTechniques for Measuring Sensemaking
and Learning within Participant-Generated Summaries. In: JASIST (accepted).
Friday, 10 May 13
44. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
Study 2:“Simplistically” measured
• If learning is closed: then a quiz
- “closed” determines WHAT should be learned
- can measure recall, but also recognising if cued by Q.
• If learning is open:
a) sub-topic count (integer) & topic quality (judged likert)
b) simple count of facts (integer) and statements (integer)
• These do not measure how “good” the learning was
Friday, 10 May 13
45. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
Study 2: Measuring “Depth” of Learning
• A theory from Education
• As learning improves
you progress up the diagram
• You begin to ‘understand’
- then critically ‘analyze’
- then ‘evaluate’ information
etc.
Image from: http://www.nwlink.com/~donclark/hrd/bloom.html
Friday, 10 May 13
46. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
Study 2: Developed 3 Scales
• 12 participants performed 3 learning tasks
- mix of high and low prior knowledge
• 1) Write summary of knowledge, 2) Learn, 3) Write summary
• 36 pairs of pre/post summaries
- 18 high prior knowledge
- 18 low prior knowledge
Friday, 10 May 13
47. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
Study 2: Developed 3 Scales
• Inductive GroundingTheory analysis
• 3 rounds of 6 high and 6 low pairs analysed by 2 researchers
• Validated by an external judge
• Until high Fleiss Kappa scores
i.e.‘substantial agreement’
Friday, 10 May 13
48. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
Study 2: Measure 1: D-Qual
scale ranging from irrelevant or useless facts (0 points) to facts that showed a level of
technical understanding (3 points). The emphasis of usefulness in this measure meant that it
was closer to the “understanding” level of Bloom’s revised taxonomy, rather than simply
“remembering”. It was important to differentiate between the two levels as many poor
summaries, as determined by the authors during the coding session, simply listed many
redundantly obvious facts (“A labrador is a dog”) rather than describing them in sentences
and summaries. For D-Qual, the judges achieved a Fleiss kappa of 0.64.
Rating Description
0 Facts are irrelevant to the subject; Facts hold no useful information or advice.
1 Facts are generalised to the overall subject matter; Facts hold little useful information or
advice.
2 Facts fulfil the required information need and are useful.
3 A level of technical detail is given via at least one key term associated with the technology
of the subject; Statistics are given.
Table 1: Quality of Facts (D-Qual).
Many of the better summaries interpreted facts into more intelligent statements. To
identify this, D-Intrp (Table 2) measured summaries in how they synthesised facts and
statements to draw conclusions and deductions (Bloom’s “analysing”) using a 3-point scale.
Measure understanding rather than remembering
Friday, 10 May 13
49. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
Study 2: Measure 2: D-Intrp
Rating Description
0 Facts contained within one statement with no association.
1 Association of two useful or detailed facts: ‘A -> B’
2 Association of multiple useful or detailed facts: ‘A+B->C’; ‘A->B->C’; ‘A->B∴C’
Table 2: Interpretation of data into statements (D-Intrp).
D-Crit reflected Bloom’s concept of “evaluating” by identifying statements that
compared facts, or used facts to raise questions about other statements. The measurement for
D-Crit was either true (1 point) or false (0 points), as shown in Table 3. A Fleiss kappa of
0.74 was achieved.
Measure analysing capabilities
Friday, 10 May 13
50. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
Study 2: Measure 3: D-Crit
Measure evaluating capabilities
Table 2: Interpretation of data into statements (D-Intrp).
D-Crit reflected Bloom’s concept of “evaluating” by identifying statements that
compared facts, or used facts to raise questions about other statements. The measurement for
D-Crit was either true (1 point) or false (0 points), as shown in Table 3. A Fleiss kappa of
0.74 was achieved.
Rating Description
0 Facts are listed with no further thought or analysis.
1 Both advantages and disadvantages listed; Comparisons drawn between items;
Participant deduced his or her own questions.
Table 3: Use of critique (D-Crit).
We did not produce a scale for level three of Anderson’s revised version of Bloom’s
taxonomy, “applying”, since the act of writing a summary would not involve the participant
to carry out a procedure that has been learned. This level of learning was thus not identifiable
in our corpus of summaries. Similarly, the highest level, “creating”, also goes beyond writingFriday, 10 May 13
51. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
Study 2: Evaluating these measures
Compare against Counting &Topic measures
measure depth (‘T-Depth’), each topic was measured on a 4-point scale ranging from not
covered (0 points) to detailed focused coverage (3 points) and averaged.
As the process of learning is primarily internal it is difficult to measure it objectively.
For this reason our measures of learning focused on the difference between pre- and post-task
knowledge held by the participant.
Code Measurement Scale
D-Qual Recall of facts 0 – 3 points
D-Intrp Interpretation of data into statements 0 – 2 points
D-Crit Critique 0 – 1 point
F-Fact Number of facts Count
F-State Number of statements Count
F-Ratio Ratio of facts per statement Average
T-Count Number of topics covered (breadth of knowledge) Count
T-Depth Level of topic focus (depth of knowledge) 0 – 3 points, averaged
Table 4: Outline of coding scheme used for analysis.
5 Results
Before beginning, the data from two participants were removed from the analysis. A
first-pass sanity check over the collected summaries revealed that they had misunderstood the
tasks set. One chose to describe their own feelings and history relating to the task topic, rather
than trying to answer the task. Another described what they intended to search for in their
• Can you differentiate pre- & post- task summaries?
• Can you differentiate high & low prior knowledge?
• How long do summaries need to be?
Friday, 10 May 13
52. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
Study 2:Analysing summaries
Pre-task example
Friday, 10 May 13
53. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
Study 2:Analysing summaries
Post-task example
Friday, 10 May 13
54. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
Study 2: Resultsknowledge, especially for pre-task summaries, which can possibly be explained that the
participants who wrote shorter summaries based on high prior knowledge are more likely to
concentrate on a single topic.
All Pre-task Post-task
D-Qual U(68) = 537.5, p = 0.32 U(34) = 125, p = 0.28 U(34) = 148, p = 0.46
D-Intrp U(68) = 642, p = 0.21 U(34) = 145, p = 0.47 U(34) = 174, p = 0.16
D-Crit U(68) = 570, p = 0.47 U(34) = 140, p = 0.47 U(34) = 144.5, p = 0.49
F-Fact t(66) = -0.4, p = 0.35 t(32) = -0.75, p = 0.23 t(32) = -0.25, p = 0.4
F-State t(66) = -0.21, p = 0.42 t(32) = -0.4, p = 0.35 t(32) = -0.17, p = 0.43
F-Ratio t(66) = 0.2, p = 0.42 t(32) = 0.31, p = 0.38 t(32) = -0.04, p = 0.48
T-Count t(66) = -0.35, p = 0.36 t(32) = 0.43, p = 0.34 t(32) = -1.01, p = 0.16
T-Depth U(68) = 721, p = 0.04 * U(34) = 194.5, p = 0.04 * U(34) = 168, p = 0.21
Table 12: Comparing high and low prior knowledge in shorter summaries. * Indicates significant results.
All Pre-task Post-task
D-Qual U(68) = 390, p = 0.01 * U(34) = 89.5, p = 0.03 * U(34) = 113.5, p = 0.18
D-Intrp U(68) = 497.5, p = 0.16 U(34) = 158.5, p = 0.29 U(34) = 95, p = 0.06
D-Crit U(68) = 693.5, p = 0.08 U(34) = 189, p = 0.05 * U(34) = 154, p = 0.32
F-Fact t(66) = 1.62, p = 0.06 t(32) = 0.64, p = 0.26 t(32) = 1, p = 0.16
F-State t(66) = 1, p = 0.16 t(32) = 0.29, p = 0.39 t(32) = 0.79, p = 0.22
F-Ratio t(66) = 0.86, p = 0.2 t(32) = 0.31, p = 0.38 t(32) = 0.21, p = 0.42
T-Count t(66) = 3.44, p = 0.0005 * t(32) = 1.92, p = 0.03 * t(32) = 2.82, p = 0.004 *
T-Depth U(68) = 572, p = 0.48 U(34) = 163, p = 0.25 U(34) = 142, p = 0.48
Table 13: Comparing high and low prior knowledge in longer summaries. * Indicates significant results.
Pretty obvious - as you can see
Friday, 10 May 13
55. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
Study 2: Results
• 1) Most measures could identify learning (between pre-post)
- more robust with longer summaries
the summaries and the prior knowledge held by the participant should be taken in to
consideration. Table 14 provides an overview of the strengths and weaknesses of each
measure and recommendations are made below. While serving as a guide readers should refer
back to the full text in our results section for more detail before using them in a study.
Identifies Learning Identifies Prior Knowledge Ignores
Length
High Low Short Long Pre Post Short Long Pre Post
D-Qual
D-Intrp
D-Crit
F-Fact
F-State
F-Ratio
T-Count
T-Depth
Table 14: Overview of measure suitability.
If participants have written shorter summaries (here averaged to around 90 words) then
learning is only really noticeable if those participants began with low prior knowledge, where
measures such as the quality of facts (D-Qual), simple fact and statement counting (F-Fact, F-
State) and topic coverage (T-Count) can be used to determine an increase of knowledge. If
short summaries are written based on high prior knowledge then only simple fact and
Friday, 10 May 13
56. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
Study 2: Results
• 2) Only some were good at identifying prior knowledge
- these required long pre-task summaries to be written
the summaries and the prior knowledge held by the participant should be taken in to
consideration. Table 14 provides an overview of the strengths and weaknesses of each
measure and recommendations are made below. While serving as a guide readers should refer
back to the full text in our results section for more detail before using them in a study.
Identifies Learning Identifies Prior Knowledge Ignores
Length
High Low Short Long Pre Post Short Long Pre Post
D-Qual
D-Intrp
D-Crit
F-Fact
F-State
F-Ratio
T-Count
T-Depth
Table 14: Overview of measure suitability.
If participants have written shorter summaries (here averaged to around 90 words) then
learning is only really noticeable if those participants began with low prior knowledge, where
measures such as the quality of facts (D-Qual), simple fact and statement counting (F-Fact, F-
State) and topic coverage (T-Count) can be used to determine an increase of knowledge. If
short summaries are written based on high prior knowledge then only simple fact and
Friday, 10 May 13
57. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
Study 2: Results
• 3) Our measures were the most robust to length of summary
- others require pushing participants beyond 200 words
the summaries and the prior knowledge held by the participant should be taken in to
consideration. Table 14 provides an overview of the strengths and weaknesses of each
measure and recommendations are made below. While serving as a guide readers should refer
back to the full text in our results section for more detail before using them in a study.
Identifies Learning Identifies Prior Knowledge Ignores
Length
High Low Short Long Pre Post Short Long Pre Post
D-Qual
D-Intrp
D-Crit
F-Fact
F-State
F-Ratio
T-Count
T-Depth
Table 14: Overview of measure suitability.
If participants have written shorter summaries (here averaged to around 90 words) then
learning is only really noticeable if those participants began with low prior knowledge, where
measures such as the quality of facts (D-Qual), simple fact and statement counting (F-Fact, F-
State) and topic coverage (T-Count) can be used to determine an increase of knowledge. If
short summaries are written based on high prior knowledge then only simple fact and
Friday, 10 May 13
58. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
Study 2: Conclusions
• We proposed a new measure based on depth of learning
- demonstrating higher levels of thinking
• This was more robust to size of written summary,
- good at long and short, while measuring learning
- able to determine if someone has existing high knowledge
• All measures did surprisingly well, for measuring learning
• Ours was most robust for determining prior knowledge level
• Future work: behaviour between good vs bad learners
Friday, 10 May 13
59. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
Talk Summary
• Search communities are trying to move beyond simple tasks
- more than result quality, and time to target
• Current focusing on understanding sessions
- which has primarily been splitting logs by time gaps
• Our work
1) moving beyond assumptions about sessions
2) introducing new methods to evaluate sensemaking
Friday, 10 May 13
60. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
Talk Summary
• There’s a long way to go before search engines know what
we’re doing beyond a query (and immediate refinements)
- there’s a long way before we do
• Also - we still need to measure:
- success in decision making (like online shopping)
- success in entertainment sessions
Friday, 10 May 13