https://dl.acm.org/citation.cfm?id=3271676 The field of Search as Learning addresses questions surrounding human learning during the search process. Existing research has largely focused on observing how users with learning-oriented information needs behave and interact with search engines. What is not yet quantified is the extent to which search is a viable learning activity compared to instructor-designed learning. Can a search session be as effective as a lecture video - our instructor-designed learning artefact - or learning? To answer this question, we designed a user study that pits instructor-designed learning (a short high-quality video lecture as commonly found in online learning platforms) against three instances of search, specifically (i) single-user search, (ii) search as a support tool for instructor-designed learning, and, (iii) collaborative search. We measured the learning gains of 151 study participants in a vocabulary learning task and report three main results: (i) lecture video watching yields up to 24% higher learning gains than single-user search, (ii) collaborative search for learning does not lead to increased learning, and (iii) lecture video watching supported by search leads up to a 41% improvement in learning gains over instructor-designed learning without a subsequent search phase.

1. Contrasting Search as Learning
Activity with Instructor-designed
Learning
Felipe Moraes, Sindunuraga R. Putra, and Claudia Hauff
@femoraesg

2. There is a shift
in learning
contexts
2

3. There is a shift in learning contexts
3
Traditional
learning contexts
Novel learning
contexts

4. Web search is a key information source
for learning
High quality learning materials
are now commonly available
online
▪ Quality in user-generated
wikis are comparable with
centrally maintained sources
▪ Instructor-designed learning
materials are often publicly
available
Nearly 30% of Web search tasks
are related to learning
▪ Education-related task
▫ Learning about a topic
▫ Learning how to do
something
4

5. Web search engines are not designed
for learning
▪ They optimise for relevance
▪ Difficult to explore a topic without prior knowledge
5
There is room for improvement!
Relevance

6. 6
People often learn together

7. [1] Meredith Ringel Morris. 2013. Collaborative Search Revisited. In CSCW ’13.
Collaborative Search as Learning
Collaborative learning
▪ Stimulates conceptual
understanding
▪ Enables co-construction of
knowledge
▪ Retains knowledge gain
▪ Promotes critical thinking
Collaborative search
▪ Improves search
effectiveness
▪ Promotes exploratory
search behaviour
▪ Is increasingly common [1]
▫ 2006: 19%
▫ 2012: 69%
7

8. Our research goal:
Explore the impact different online learning
materials have on learning gain
8

9. 9
SearchX: our open source collaborative search system
http://felipemoraes.github.io/searchx

10. User study overview
10
151
Participants
Collaborative
Search (CSE)
Single-User
Search (SE)
Video (V) Video +
Search (V+S)
Video +
Search
20min
(V+S20)
£5.00 per hour
27 days

11. Data collected
11
Learning Task:
Learn more information
regarding a specific
(learning) topic
Interaction Logs
Assessment
Search
Behaviour
Learning
Gain

12. Learning Task Topic
12
Topic: Radioactivity decay
The professor requires all students to demonstrate what
they learn about a particular topic ...
... collect and save all the web pages, publications, and other
online sources that are helpful .
... you will be asked to complete 13 exercises; those
exercises include questions [...]
Vocabulary:
Auger electron, Coulomb barrier,
beta-plus emission, isomeric transition
K-shell electron, isobaric process,
beta-minus decay, conversion electron,
antimatter, electron capture decay

13. [1] Edgar Dale. 1965. Vocabulary measurement: Techniques and major findings. Elementary English
Vocabulary Knowledge Scale (VKS)
A reliable indicator of vocabulary knowledge [1]
13
1. I don’t remember having seen this term/phrase before.
2. I have seen this term/phrase before, but I don’t think I know what it
means.
3. I have seen this term/phrase before, and I think it means ___.
4. I know this term/phrase. It means ___.
5. (Not used) I can use this term/phrase in a sentence: ___.

14. Learning Assessment
14
Pre-test
- 10 questions on topic-specific vocabulary (e.g. auger
electron)
- Multiple choice using Vocabulary Knowledge Scale (VKS)
Learning phase
Post-test

15. Example of a search as learning session
15
what is radioactivity → types of radioactivity → how does radioactivity occur
→ radioactivity resources online → uses for radiation → university resources radioactivity
→ radioactivity explained → electromagnetic radiation → radioactivity harmful
→ ionising and non ionising radiation → radioactivity studies physics journals
→ radioactivity exposure → what creates gamma waves → xray radiation type
→ dangers of radioactivity → uses of radioactivity

16. Learning material selection
16
Most
Popular
30
Most
Difficult
Topic Video length
Radioactive decay 6m53s
Qubit 12m24s
Water quality aspects 10m45s
Religions 11m09s
Sedimentary rocks 5m03s
Anesthesia 4m55s
Glycolysis 13m29s
Urban water cycle 7m40s
Depression 4m28s
Industrial biotechnology 5m48s

17. Measuring learning gain
17
Aggregate difference in knowledge
observed after and before the
experiment.
Absolute Learning
Gain (ALG)

18. Our research questions
▪ RQ1 How effective (with respect to learning outcome) is searching to
learn compared to instructor-designed learning?
▪ RQ2 How effective (with respect to learning outcome) is
instructor-designed learning supported by search in comparison to
just instructor-designed learning?
▪ RQ3 How effective is pair-wise collaborative search compared to
single-user search for learning?
18

19. Results: RQ1
Search vs. instructor-designed learning
19
SE V V+SE V+SE20
ALG 0.281 0.320 0.420 0.444
Trend: instructor-designed learning leads to a 14%.

20. Results: RQ1
Search vs. Instructor-designed Learning
20
Participants in the SE condition
have the largest percentage (40%)
of vocabulary items that remain
at knowledge levels 1/2.
For the SE condition on average
participants reached levels of 3/4
for 30% of the terms unknown to
them.

21. Results: RQ3
Collaborative Search As Learning
21
SE V V+SE V+SE20 CSE
ALG 0.281 0.320 0.420 0.444 0.234
Collaborative search participants performed
significantly worse than participants in the
video-only condition.

22. Results: RQ3
Collaborative Search As Learning
22
Participants with an
undergraduate degree show
consistently higher gains than
participants with a high-school
diploma

23. Concluding
▪ We ran a large-scale user study and measured the learning
gain of users using different source of learning materials
▪ Participants with instructor-designed materials reached
higher learning gains than participants using only search for
learning
▪ Instructor-designed materials supported by search is
superior to instructor-designed materials alone
▪ Collaborative search did not reach higher gains than
single-user search
23

24. Next steps
▪ Experiment with a more complex learning task
▪ Explore sensemaking interfaces
▪ Move SearchX from the lab setting into a MOOC
▪ Investigate the failure of collaborative search for learning
24

25. Thanks!!@femoraesg
f.moraes@tudelft.nl
25
http://felipemoraes.github.io/searchx