In this presentation I illustrate the methodology used to measure the relationship between student attainment, engagement, and self-efficacy beliefs through Student Response Systems. https://sites.google.com/site/fabioarico

1. 1
When Student Confidence Clicks
Academic Self-Efficacy
and Learning in HE
Fabio R. Aricò
Using SRS (clickers)
methods, analysis,
and results

2. OUTLINE
1. Student Response Systems (clickers)
2. Use of clickers in “Introductory Economics”
learning environment & learning technology
3. Collecting data and analysing results
• demographic variables, data coding, diagrams
• Result 1: assessing self-assessment  the role of clickers
• Result 2: peer-instruction and learning-effects
2

3. ETHICAL REMARK
You will be presented with data collected during teaching sessions.
Students involved have given informed consent for me to analyse
their responses and present the results of this analysis.
I can assist with ethical queries as well, please ask me.
3

4. 1. Using Student Response
Systems (clickers)
4

5. Do you know clickers? Do you know how to use them?
A. Yes, I use them in my teaching.
B. Yes, but I do not currently use
them in my teaching.
C. I have an idea of what clickers
do, but a very basic one.
D. I have no clue about what
clickers do and how they
can be used.
0% 0% 0% 0%
5
Yes, I use them in my tea...
Yes, but I do not currentl...
I have no clue about what...
I have an idea of what cli...

6. 6
SRS interaction with feedback to the students

7. CLICKER MANAGEMENT
School of Economics Protocol:
1. Each student receives a clicker during orientation week.
Information sheet. Clicker collection is not compulsory.
2. Clicker ID associated to Student ID
Ethical procedure followed – Informed consent to use data.
3. Students use clickers across their 3 years of study and then
return their clicker at the end of their studies to the School.
4. If a student loses/breaks his/her clicker, s/he has to pay a
replacement cost (but can obtain a new one).
7

8. CLICKER MANAGEMENT
Statistics for 2013-14 - Year 1
180 students enrolled
13 students did not collect their clickers
(majority of overseas students)
1 clicker lost paid for and replaced
1 clicker returned (after chasing)
1 clicker missing
8

9. CORE IDEAS ABOUT CLICKERS
• Clickers increase engagement and student satisfaction
 Yes, it works and it is tested.
• The ‘clicker’ novelty wears out quickly
 Possibly, but (majority) students keep on using them and enjoy them.
• Cannot be over-ambitious in what you teach
 If you switch to clickers you will need to cut some of the material.
• Use a counter to get responses in faster
 Recent research suggests 80% response rate is the cut-point.
• It is not all about technology, it is still about good teaching
 See Nielsen et. al. (2013) , “Research in Learning Technology”, ALT
9

10. 2. “Introductory Economics”
The learning environment
10

11. TEACHING PROTOCOL – the module
Introductory
Macroeconomics  Year 1 – compulsory year-long module - 180 students
Lectures  traditional frontal-teaching (10 per sem.)
Seminars  small group, pre-assigned problem sets (4 per sem.)
Workshops  large group, problem-solving sessions (4 per sem.)
Support Sessions  non-compulsory drop-in sessions (4 per sem.)
11

12. TEACHING PROTOCOL – lectures
Lectures  interaction via clicker technology
Seminars  revision questions + understanding questions
Workshops  closing questions:
was the lecture enjoyable interesting?
was the material difficult?
Support Sessions  online report of clicking session + feedback
12

13. Lecture difficulty indicator -
66% (+8%).
Please look out for additional
resources coming online very
shortly. Video tutorials about
the IS-LM will be available
shortly.
I would like you to reflect on
the feedback asked on the IS-LM
model and try to identify
what are your OWN
difficulties. If many of you are
confident about
understanding and mastering
the material, we need to make
this belief becoming a reality.
For those of you who are not
confident. Why is this the
case? Come and discuss this
with me.
13

14. TEACHING PROTOCOL – lectures
14
100.0%
90.0%
80.0%
70.0%
60.0%
50.0%
40.0%
30.0%
20.0%
10.0%
0.0%
Longitudinal Lecture Assessment
2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21
Lecture Week
easy
enjoy
attended
revision

15. TEACHING PROTOCOL – seminars
Seminars  preliminary Seminar Quizzes (paper-based)
Seminars  3-4 revision/understanding questions
Workshops  2 confidence/self-assessment questions
Sessions  open-answer comments
Support Sessions  online report of Seminar Quiz
- solutions and overall performance
- individual performance available
- response to open-answer comments
15

16. TEACHING PROTOCOL – seminars
16

17. TEACHING PROTOCOL – workshop
Workshops  peer-instructed flipped classroom approach
Seminars
17

18. If the bargaining power of trade unions increases…
A. Unemployment will
increase.
B. Unemployment will
decrease.
C. Unemployment will stay
the same.
18

19. How confident do you feel about your answer?
A. Very confident.
B. Confident.
C. Somewhat Confident.
D. Not Confident.
19
27%
33%
13%
27%
A. B. C. D.

20. So did you get the right answer?
Compare your answer with the delegates
sitting next to you for a minute and then
we will try to re-poll the question.
20

21. If the bargaining power of trade unions increases…
A. Unemployment will
increase.
B. Unemployment will
decrease.
C. Unemployment will stay
the same.
21
40%
30% 30%
A. B. C.

22. If the bargaining power of trade unions increases…

23. TEACHING PROTOCOL – workshop
Workshops  standard algorithm:
1. Preliminary preparation question
2. Quiz questions + Confidence questions (no solution)
2. Peer-instruction learning
3. Quiz questions + solutions
4. Problem-set questions (no clicking)
4. Feedback questions:
- what was the cause of mistakes/problems?
- did you enjoy using clickers?
- were clickers useful to your learning?
Support Sessions  online report of clicking session + feedback
23

24. TEACHING PROTOCOL – extra-curricular
Extra-Curricular  Activities to promote engagement and Self-Efficacy
Seminars  Module Facebook Page + Blackboard pages
- ‘challenges’ to encourage further study
- interaction and participation
Seminars  Voluntary in-lecture presentations (5 minutes)
- to exploit demonstration effects
Support Sessions  Campus Vouchers (for engagement, not attainment)
24

25. 3. Collecting data and
analysing results
25

26. TEACHING PROTOCOL – the methodology
Targets  attainment, engagement, academic self-efficacy
role of the SRS (clicker) technology
Learning analytics  rich dataset = clicker and paper-based responses
Seminars  matched demographics from student records
 uncover correlation patterns
Qualitative data  focus group and individual interviews
Sessions  feedback from students
Support Sessions  provide the narrative to interpret the analytics
26

27. EVIDENCE-BASED PRACTICE
27
ATTAINMENT
Gender
Satisfaction
ENGAGEMENT CONFIDENCE
Previous
Domicile attainment POLAR
Perceived
difficulties

28. DATASETS
Student Q1 Q2 Q3 … 
1 0 1 1
2 1 0 0
3 1 1 …
…
 performance per question
confidence by question
performance per student
confidence by student
longitudinal study
- across all lectures
- across all seminars
- across all workshops
Intermediate and final
attainment outcomes
- course test
- final exam
28

29. 3. Collecting data and
analysing results
Result 1: Peer-instruction and learning
29

30. RESULT 1: Peer-instruction and learning
30
We have analysed the teaching protocol adopted in workshops
1. Ask a question (do not show distribution of answer)
and poll it through clickers.
2. Ask students to self-assess the correctness of their answer:
how confident are they about having given the right answer?
3. Ask the same question again, show the distribution of answers.
Reveal the correct answer and teach why this is the case.
The difference between % correct answers in the two rounds
represents the learning gain generated by peer-instruction.

31. RESULT 1: Some ideas to represent data
31
Week 5 (2012-13)
% correct responses
■ 1st round
■ 2nd round
You can represent
results by question…

32. RESULT 1: Some ideas to represent data
% correct responses 1st round
% correct responses 2nd round
32
Week 5 (2012-13)
You can represent
results by student…

33. RESULT 1: Some ideas to represent data
Workshop Longitudinal
data 2013-14
33
70%
60%
50%
40%
30%
20%
10%
0%
1 2 3 4 5 6 7
Workshop Number
prepared
Q.Score
R.Score

34. RESULT 1: Some ideas to represent data
Workshop Longitudinal
data 2013-14
34
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
1 2 3 4 5 6 7
Workshop Number
Q.Score
R.Score
SelfAss
CorrConf

35. RESULT 1: Some ideas to represent data
Workshop Longitudinal
data 2013-14
35
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
MYFAULT
ENJOY
LEARN
1 2 3 4 5 6 7

36. RESULT 1: Peer-instruction and learning
36
We found that the learning gains from peer-instruction:
• are higher in the group of low-performing students
 so peer-instruction works in getting everybody at the same level.
• are not associated to self-efficacy and self-assessment skills
 so everybody has a chance to learn despite being poor at self-assessing.
• are positively correlated to exam performance
 so it seems that peer-instruction can display some long-run effects.
 This result has to be investigated with further attention.

37. 3. Collecting data and
analysing results
Result 2: Assessing Self-Assessment
37

38. RESULT 2: Assessing Self-Assessment
38
What is the relationship between attainment and confidence?
Are students able to self-assess their performance?
Compare two self-assessment set-ups:
• Seminars paper-based self-assessment
seminar quizzes + confidence questions
• Workshops clicker-based self-assessment
peer-instructed re-iterated algorithm

39. RESULT 2: Seminars
What is the relationship between attainment and confidence?
39

40. RESULT 2: Seminars
What is the relationship between attainment and confidence?
40

41. RESULT 2: Workshops
What is the relationship between attainment and confidence?
41

42. RESULT 2: Workshops
What is the relationship between attainment and confidence?
42

43. RESULT 2: Summary
• In Seminar Quizzes:
 high-attainment students display higher confidence
 low-attainment students not able to self-assess their performance.
• In Workshop sessions:
 high-attainment students display higher confidence
 low-attainment display lower confidence.
• How to interpret this asymmetry?
43

44. RESULT 2: Summary
• In Seminar Quizzes:
 3 or 4 questions, paper-based quiz, 5-6 minutes, not anonymous
 1 confidence assessment for overall performance.
• In Workshop sessions:
 5-10 questions, clicker response, slower pace, quasi-anonymous
 1 confidence assessment for each question asked.
44

45. RESULT 2: Conclusion
• Low-attainment students encounter more difficulties in
self-assessing their performance in an environment where:
 they self-assess their ‘overall’ performance on a composite task
 they are exposed to questions for a shorter period of time
 they are exposed to fewer questions, not anonymously.
• Focus group interviews (differentiated by attainment groups)
confirm that low-attainment students display lower
self-assessment skills  important finding for intervention!
45