Measuring Self-Regulated Learning

1. MEASURING SELF-REGULATED LEARNING
Piia Näykki
Learning and Educational Technology Research Unit
1

2. OUTLINE OF THE LECTURE
1. Key points of SRL and measurement
2. Research protocols that are currently used to measure SRL
3. Constructive critique for the measures
4. Primary vs. secondary research
5. Forecast what measurements of SRL might be in the future
27.11.2015
2

3. KEY POINTS IN SRL
The general agreement in the research field:
• The term SRL is associated with forms of learning that are
metacognitively guided, intrinsically motivated, and strategic
• There are a number of SRL models
• SRL models emphasizes different phases
• SRL is a cyclical process
3
This is what we already know!
hmmmmm.. But how to study SRL?

4. 4
BRAINSTORM in pairs (5 min):
1. Think, what do you allready know
about Self-Regulated Learning and
2. What aspects of SRL you would be
interested to study?
3. Formulate a list of interesting
questions related to SRL.
4. Prepare yourself to share your ideas to
the class.

5. 5
SHARE your ideas to the class,
As we continue the lecture, think following
question:
- How could this topic of my interest be
studied?

6. KEY POINTS OF MEASUREMENTS OF SRL
1. Measurement of SRL should reflect a model of SRL.
2. From the description of SRL it is obvious that many facets of
SRL are not readily observable.
3. One challenge of studying SRL is to find ways to document its
components.
4. How to study SRL as an aptitude and/ or as an event?
27.11.2015
6

7. SRL AS AN APTITUDE… AS AN EVENT
1. When SRL is measured as an aptitude, a single measurement
aggregates over or abstracts some quality of SRL based on
multiple SRL events.
Example:
- student studying for a test. A researcher may be interested in mc
monitoring concerning a rehearsal tactic.
- The student’s description of SRL might be recorded in several
forms: a rating on a questionnaire item, an interview’s
classification or the proportion of the particular kinds of notes
student has written to the textbook.
- To predict whether a student will/ will not, can/ can not act on an
SRL-related cognition.
27.11.2015
7

8. SRL AS AN APTITUDE… AS AN EVENT
1. When SRL is measured as an event has 3 levels: occurency,
contingency and patterned contingency.
Example:
- Student is solving a geometry problem and thinking aloud.
- The student says: ”wow, this is hard”.
- The researcher may presume that s/he have monitored the
current state of the task (task difficulty).
- What are the standards against this monitoring action has been
done? (we don’t know).
- Student’s report is interpreted as indirect evidence.
27.11.2015
8

9. WHY IT IS SO DIFFICULT TO MEASURE SRL
• SRL is an internal process
• Can we ask individuals to tell their internal processes?
• What problems are related to that?
• SRL has an external signs,
• What kind of signs?
• Are we sure that our interpretations of the signs are valid?
• What problems are related to that?
9

10. TENSIONS IN MEASURING SRL
• General Reliability & validity
• Big sample size vs. sample with a meaning
• Self-report critique: Self-report is too static and ”Individuals
tend to tell what they expect researchers want to hear”.
• Qualitative approaches critique: video observation methods are
subjective and based on researchers’ interpretation, not
objective.
• In sum: Some of the features of the classical
QUANTITATIVE VS. QUALITATIVE divide
10

11. OUTLINE OF THE LECTURE
1. Key points of measurements in general
2. SRL: components researchers and practioners seek to
measure
3. Research protocols that are currently used to measure SRL
4. Constructive critique for the measures
5. Primary vs. secondary research
6. Forecast what measurements of SRL might be in the future
27.11.2015
11

12. PROTOCOLS FOR MEASURING SRL
Winne & Perry; Boekaerts & Corno (2005) + Handbook2011
AS AN APTITUDE
• Self-report questionnaire
• Structured interviews
• Teacher judgements
AS AN EVENT
• Observations of overt behavior (video/ real time)
• Thinking aloud protocols
• Traces of mental events and processes (computer)
• Learning diaries
• Interviews (possible with a stimulus)
12

13. 13
BRAINSTORM in pairs (5 min):
1. Take your list of interesting questions
related to SRL.
2. Now think and discuss in pairs: how
could your topic of interest be studied?
3. Prepare yourself to share your ideas to
the class.

14. OBSERVATIONS OF OVERT BEHAVIOR
• The famous Marshmallow experiment
http://www.youtube.com/watch?v=QX_oy9614HQ
• 1960’s Standford University
• 4 year old kids, 15 minutes to waite
• 2/3 could not waite..
• 1/3 – delayed gratification
• 14-15 years later follow-up studies with the same kids.
• Around 100% of those who waited were successful in school
• Around 80% of those who could not waite were in troubles.
27.11.2015
14

15. OBSERVATIONS OF OVERT BEHAVIOR
• Capture what the learner is doing
• Based on verbal (talk), actions (gestures) and interactions
• Use a coding categories (pre-defined or based on the data)
• Recording for inter-judges reliability
Problems:
• Are we really accessing the students SRL?
• Do we make inferences as researchers?
• How does a SRL ”look like”? How it can be recognized? How to
operationalize the phenomenon?
27.11.2015
15

16. THINKING-ALOUD PROTOCOLS
Example of thinking-aloud (02:45-03:40)
http://www.youtube.com/watch?v=px0A4jV02RA&feature=relate
d
• Students say out loud what they are doing, thinking, etc.
• Ericsson, K. A., & Simon, H. A. (1980)
• Coding and inter-judges
• Researcher presence should not affect but activate
• Direct measure on-task
• Problems: requieres training, reliability with younger children,
task difficulty, high cognitive load, frequency affected by
personal factors.
27.11.2015
16

17. DIARIES
• Students report in a scheduled basis their strategy use and
problems
• Bernhard Schmitz & Julia Klug (Handbook, 2011)
• Qualitative but also quantitative (structured questions)
• Time series analysis
• Type of Self-report
• Problems: Reactivity, high level of students implication
27.11.2015
17

18. INTERVIEW
• Asking questions about individuals’ self-regulated learning
• Interview can be unstructured, semi-structured or structured
• Stimulated recall interview
• Participants explain what was happening in a certain situations
(i.e. in the video).
• Coding needed: Inter-judges
• Problems: off-task, students’ accuracy, laborous: transcriptions
needed.
27.11.2015
18

19. TRACES OF MENTAL EVENTS & PROCESSES
Example: Eye-tracking experiment, online dating
http://www.youtube.com/watch?v=WqD2pXqT0Z0
• Follow the students actions called traces.
• Students behavior (Computer technology based)
• Examples: gStudy & nStudy (Winne et al.)
• Objective collection
27.11.2015
19

20. LOGFILE TRACES
• Time stamped records of learners activity in computer based
learning environment.
• Provide data ”on the fly” of learners actions.
• Possible to capture in details each action lerner produces (Perry
& Winne, 2006).
• Questionnaires and self-reports do not reliably answer what the
students actually do (Hadwin et al., 2002).
• Depending on the age group, often children cannot destinquish
they actions from actual behavior (Paris & Paris 1990).

21. WHEN THERE IS A NEED TO USE ON-LINE
METHODS
• Usability problems
• Learner – hypermedia interaction, such as navigation
• Modeling learners actions
• Investigation of cognitive processes during the learning
• To better capture intra-individual differences on learners
actions
(Rouet & Passerault, 1999; Winne 2001; Hadwin et al., 2007)

22. DURATION, FREQUENCY AND SEQUENCY OF EVENTS
• Duration = how long time something is happening
• i.e. duration of reading (eg. Time spent in reading)
• duration of model events (eg. Time spent in processing or
manipulating the information).
• Frequency = how often a certain event occurs?
• i.e. amount of checking the instructions.
• Sequency = how the events are sequenced.
• i.e. sequency of the specific events occuring.
• Data parsing to identify the meaningful events from the
extagenous events that occur in the data (Nesbit et al., 2008).
XWSAJNSHBHCLSOD
XWSAJNSHBHCLSOD -> ABC

23. • gStudy is designed in order to
supports students to became and
practise self-regulated learning
skills (Winne et al., 2005).
• gStudy provides a number of
cognitive tools, for example
highlighting with different labels,
creating notes that prompts to
elaborate or summarize or making
searches from the contents of the
current kit (Kumar, Groeneboer, Chu,
Jamieson-Noel. & Xin, 2006; Winne & al., 2006).
• Students cognitive tool use in
gStudy is considered to reflect
students use of study tactics (Hadwin
et al., 2008)
GSTUDY

24. LEARNING PATTERNS IN CHALLENGING LEARNING
SITUATIONS
•In the challenging learning situations, 88 learning patterns
emerged between the low achieving students.
•These patterns varied from lenght of three to eight.
•Each pattern was used eight to twelve times between
challenging gStudy sessions.
2
1 1
2
1
2 2 2 2 2
1
1 2 3 4 5 6 7 8
Label: Interesting information
Label: Important information
1 11
3 3
1 2 3 4
Label: Interestimg detail
Make note in C-map

25. nStudy
Winne & Hadwin, 2009

26. TRACES PROBLEMS
• Problems: students can use actions for different purposes
(researcher inferences), Massive data: analysis problematic
26
27.11.2015

27. SELF-REPORT
• A self-report inventory is a type of psychological test in which
a person fills out a survey or questionnaire with or without the
help of an investigator
• Likert-scales (5 or 7)
• Quantitative data
• Open questions could be qualitative
• Large sample size
• Easy interpretation
• High reliability if well-constructed
• MSLQ, LASSI, OSLQ, ILS…
27.11.2015
27

28. SELF-REPORT PROBLEMS
• Validity
• Honesty & accuracy (Introspective ability)
• Decontextualized
• Too general if not tailored
• In sum: Like other methods have problems
27.11.2015
28

29. MIXED METHODS TO STUDY SRL
• Triangulation
• Validity
• Different information: bigger picture
• Example of combinations:
• Self-report with other
• TAP + stimulated recall
• Traces + stimulated recall
27.11.2015
29

30. RESEARCH EXAMPLE
Mixed methods approach: video data analysis of group interaction and
video-stimulated recall interview.
Näykki, P. Järvelä, S. Kirschner, P., & Järvenoja, H. (2014). Socio-emotional conflict in collaborative
learning – A process-oriented case study in a higher education context. International Journal of
Educational Research, 68, 1-14.

31. How did the students interpret the conflict?
Emma and Anna reacted the most strongly to the conflict
EMMA: ‘It was a huge shock to me; I was so surprised that
strange people can talk to each other like that. So I shut
down, and I thought, “Oh my god, can I say what I think
at all?”
ANNA: ‘I think we were overruled, and I didn’t enjoy the
group work after the conflict; it was just to get the
course done. At first, I tried to negotiate the task with
Erik, but in the end, I took a yes-man role and tried not
to care and to agree on every solution’.

32. Erik and Maria explained the conflict more neutrally
ERIK: ‘The personal chemistry didn’t work between two of the
girls in our group, Tiina and Maria. Actually, there were three
central characters, me, Maria, and Tiina, and then the two
that were a bit quieter, Anna and Emma. Very clearly, Maria
was bothered of Tiina’s occupation; I think she was jealous of
her and was making fun of her’. Maria andTiina had this fight, and
after that, everything was kind of hushed up and—I don’t know—I
sometimes felt that I was some kind of a leader; others wanted me to
comment on everything’. My goal was to learn as much as possible because
this whole area is so interesting and very valuable to me’. But after the
conflict, it was more like, let’s just try to do the task, and we just tried to get
group work done as soon as possible’.

33. KEY POINTS OF MEASUREMENTS IN GENERAL
• Reliability:
• Overall consistency of a measure: Does it measure what we are
aiming for?
• Inter-judges reliability.
• Validity:
• Are these results meaningful: How to interpret the findings?
• Internal validity: extent to which a causal conclusion based on a
study is warranted. Is my study valid? Did I interpret the results
correctly?
• External validity: extent to which the results of a study can be
generalized to other situations and to other people. Does this
happen in the real world?
33

34. OUTLINE OF THE LECTURE
1. Key points of measurements in general
2. SRL: components researchers and practioners seek to
measure
3. Research protocols that are currently used to measure SRL
4. Constructive critique for the measures
5. Primary vs. secondary research
6. Forecast what measurements of SRL might be in the future
34

35. PRIMARY VS. SECONDARY RESEARCH
• Primary research consists of the collection of original primary
data by the researcher. It is often undertaken after the
researcher has gained some insight into the issue by reviewing
secondary research or by analyzing previously collected primary
data.
• Secondary research (also known as desk research) involves the
summary, collation and/or synthesis of existing research rather
than primary research, where data is collected from, for
example, research subjects or experiments.
35

36. TYPES OF SECONDARY RESEARCH
• Different types
• Systematic Review (Meta-Analysis)
• Best-Evidence Synthesis
• Narrative Review
• Different Objectives
• Integrative Research Review
• Theoretical Review
• Methodological Review
• Thematic Review
• State-Of-The-Art Review
• Historical Review
• Comparison of two perspectives review
36

37. META-ANALYSIS
• A meta-analysis refers to:
• Methods focused on contrasting and combining results from
different studies…
• …in the hope of identifying patterns among study results…
• …giving statistics to evaluate the effect of the relationships.
• It uses the different studies’ effect sizes comparing among
them taking into account other variables (e.g. sample size) to
weight the importance of a particular study for the general
research aim.
37

38. CRITIQUES TO META-ANALYSIS
• A meta-analysis of several small studies does not predict the
results of a single large study (medicine).
• Golden standard Randomized Controlled Trial vs. meta-
analysis –RCT is just one study.
• Publication bias: the file drawer problem.
• Different study goals, methods, etc.: Mixing apples and
oranges
• Inclusion criteria is CRUCIAL: Garbage in, garbage out
• In sum: there are bad meta-analysis as much as there are
excellent narrative reviews. MA per se does not guarantee
quality.
38

39. META-ANALYSIS AND SRL
SRL relatively a new field
• Four MAs
• Hattie, Biggs, & Purdie, (1996).
• Dignath, & Büttner, (2008).
• Sitzmann, & Ely, (2011).
• Dignath, Büttner, & Langfeldt, (2008).
39

40. DIGNAGTH’S MA FEATURES
• Goal: reviewed 48 studies derived from 30 articles on the
effectiveness of self-regulatory training with primary school
students.
• Detailed eligibility criteria.
• SRL training programmes have a positive effect on learning,
strategy use & motivation, even for primary school children
• Highest benefits can be gained in mathematics, motivational
outcomes, and cognitive & metacognitive strategies (in line
with Hattie et al.,1996).
• Primary students benefit even more than older (in line with
Hattie et al.,1996).
40

41. DIGNAGTH’S MA FEATURES
• Surprisingly, effect sizes were significantly higher for
interventions that did not train students by means of group
work but…
• Several meta-analyses investigated this method and revealed
positive effects:
• In the studies included in this meta-analysis, we found only very
little information about the implementation of group work in the
learning setting.
• Hence, a possible reason for the negative effect of group work on
training effects at primary school level might be that students
were not used to work in groups and did not receive enough
instruction about collaboration.
• Too many self-report data (2006)
41

42. DIGNAGTH’S MA FEATURES
• Summarizing the most effective characteristics of interventions
yields that a training programme should be based on social-
cognitive theories, should train cognitive (especially
elaboration and problem solving strategies), metacognitive
(especially planning strategies), and motivational strategies
(especially feedback), and provide knowledge about strategy
use and about its benefit.
• Future research: learning environment & how teachers can be
trained.
• PREP EXAMPLE (Preparing Teacher Education Students for 21st Practice Learning)
42

43. 43
BRAINSTORM in pairs (5 min):
1. Take your list of how to study (a certain
topic of) SRL
2. Now think and discuss in pairs: what
do you think about measuring SRL
now?
3. Prepare yourself to present 3 key
points to the class.

44. FORECASTING THE FUTURE
11/27/2015
44

45. FORECASTING THE FUTURE
11/27/2015
45
• Too little has been achieved yet in measuring SRL as an event.
• Challenges in this arena are significant!
• Protocols are needed for collecting longitudinal measurements
that span multiple brief episodes as well as expended periods
(i.e. grade levels).
• Methods are needed that characterize temporally unfolding
patterns of engagement with tasks
• in terms of the tactics and strategies that constitute SRL.
• In terms of comparing patterns over time
• New ideas from psychophysiological measurements
• Triangulation needed!

46. CONCLUSIONS
Different ways to measure
• Choose the one that fits your research goals.
• But don’t avoid because lack of knowledge.
• Try to triangulate data
• Don’tbe afraid to be innovative but remember:
• Reliability and validity
27.11.2015
46

47. CONCLUSIONS
1. Measuring intervenes in a student’s environment.
• We design measurement with an intention to cause the student to
recall or to generate a particular kind of response.
WHEN WE MEASURE, WE CHANGE THE ENVIRONMENT – ALLWAYS.
27.11.2015
47

48. REFERENCES
• Boekaerts, M., & Corno, L. (2005). Self-regulation in the classroom: A perspective on assessment and intervention.
Applied Psychology-an International Review-psychologie Appliquee-revue Internationale, 54(2), 199-231.
• Butler, D. L. (2011). Investigating self-regulated learning using in-depth case studies. In B. J. Zimmerman & D. H.
Schunk (Eds.), Handbook of self-regulation of learning and performance(pp. 346-360). New York: Routledge.
• Efklides, A. (2011). Interactions of metacognition with motivation and affect in self-regulated learning: The MASRL
model. Educational Psychologist, 46(1), 6 -25. doi: 10.1080/00461520.2011.538645
• Ericsson, K. A., & Simon, H. A. (1980). Verbal reports as data. Psychological Review, 87(3), 215-251. doi:
10.1037/0033-295X.87.3.215
• Samuelstuen, M. S., & Bråten, I. (2007). Examining the validity of self-reports on scales measuring students' strategic
processing. British Journal of Educational Psychology, 77(2), 351-378. doi: 10.1348/000709906x106147
• Schmitz, B., Klug, J., & Schmidt, M. (2011). Assessing self-regulated learning using diary measures with university
students. In B. J. Zimmerman & D. H. Schunk (Eds.), Handbook of self-regulation of learning and performance(pp.
251-266). New York: Routledge.
• Winne, P. H., & Hadwin, A. F. (1998). Studying as self-regulated engagement in learning. In D. Hacker, J. Dunlosky &
A. Graesser (Eds.), Metacognition in educational theory and practice(pp. 277-304). Hillsdale, NJ: Erlbaum.
• Zimmerman, B. J., & Moylan, A. R. (2009). Self-regulation: Where metacognition and motivation intersect. In D. J.
Hacker, J. Dunlosky & A. C. Graesser (Eds.), Handbook of Metacognition in Education(pp. 299-315). New York:
Routledge.
• Zimmerman, B. J., & Schunk, D. H. (2011). Handbook of self-regulation of learning and performance. New York:
Routledge.
27.11.2015
48

49. REFERENCES
• Cooper, H. (2010). Research synthesis and meta-analysis. Thousand Oaks, California: SAGE.
• Dignath, C., & Büttner, G. (2008). Components of fostering self-regulated learning among students. A meta-analysis
on intervention studies at primary and secondary school level. Metacognition and Learning, 3, 231-264. doi:
10.1007/s11409-008-9029-x
• Dignath, C., Büttner, G., & Langfeldt, H. (2008). How can primary school students learn self-regulated learning
strategies most effectively? A meta-analysis on self-regulation training programmes. Educational Research Review,
3(2), 101-129. doi: 10.1016/j.edurev.2008.02.003
• Dochy, F. (2006). A guide for writing scholarly articles or reviews for the Educational Research Review. Educational
Research Review. http://www.journals.elsevier.com/educational-research-review/
• Hattie, J., Biggs, J., & Purdie, N. (1996). Effects of learning skills interventions on student learning: A meta-analysis.
Review of Educational Research, 66(2), 99-136. doi: 10.3102/00346543066002099
• Sitzmann, T., & Ely, K. (2011). A meta-analysis of self-regulated learning in work-related training and educational
attainment: What we know and where we need to go. Psychological Bulletin, 137(3), 421-442. doi: 10.1037/a0022777
27.11.2015
49

50. THANK YOU!
QUESTIONS AND COMMENTS?
piia.naykki@oulu.fi
50