The document discusses the automation of feedback and assessment in programming education, specifically through the use of a system called Weblab at TU Delft. It highlights the challenges of grading traditional paper exams and proposes an integrated learning management system that facilitates automatic grading, immediate feedback, and continuous monitoring of student progress. The research agenda includes exploring the applicability of automated assessment in various programming courses and addressing issues like plagiarism and learning analytics.

1. Automating Feedback & Assessment
in WebLab
Eelco Visser
TU Delft | Open Education Week 2015 | Research Seminar: Data Science | March 9, 2015

2. 2010-2011

3. Paper Exams on Programming
Tedious to grade!
Is this a correct solution?
Can you grade that objectively?
Is programming on paper a
useful skill?
x 5 pages x 150 students

4. 31 B A D C
32 C D A B
33 D C B A
34 A D C B
35 D A B C
36 B C D A
37 C B A D
38 A C D B
39 C A B D
40 B D C A
41 D B A C
42 A C B D
43 C A D B
44 B D A C
45 D B C A
46 C B D A
47 A D B C
48 D A C B
49 A B D C
50 B A C D
51 C D B A
52 D C A B
53 A B C D
54 B A D C
55 C D A B
56 D C B A
57 A D C B
58 D A B C
59 B C D A
60 C B A D
niet inkleuren a.u.b.
Handtekening
Gemaakt door:
TU Delft
Vakcode:
Dag Maand Jaar
/ /
Naam:
Tentamen:
In BLOKLETTERS invullen a.u.b.
16 C A D B
17 B D A C
18 D B C A
19 B C A D
20 C B D A
21 A D B C
22 D A B C
23 A B D C
24 B A C D
25 C D B A
26 D C A B
27 A B C D
28 B A D C
29 C D A B
30 D C B A
61 C D B A
75 A C B D
62 D C A B
74 D B A C
63 A B C D
73 B D C A
64 B A D C
72 C A B D
65 C D A B
71 A C D B
66 D C B A
70 C B A D
67 A D C B
69 B C D A
68 D A B C
Vierkeuze antwoordformulier FB1
dag
avond
Vakje volledig invullen
1 D A B C
2 B C D A
3 C B A D
4 A C D B
5 C A B D
6 B D C A
7 D B A C
8 A C B D
9 C A D B
10 B D A C
11 D B C A
12 B C A D
13 C B D A
14 A D B C
15 D A C B
StudentNummer
0
1
2
3
4
5
6
7
8
9
Vakje volledig invullen
Opmerkingen:
4124
A: {contents, i} B: {contents} C: {that} D: {that, i}
Question 9 (Recursion): Consider the following function f
def f(a: Int, b: Int): Int = if (b == 0) a else f(b, a % b)
In executing the call f(30,4), how many times is f called?
A: 1 B: 2 C: 3 D: 4
Question 10 (Interpreter): Given the definitions in Figure 3, what is the value of the
following expression?
eval {
Let("x", Number(1),
Let("f", Fun("y", Plus(Var("x"), Var("y"))),
Let("x", Number(2),
App(Var("f"), Number(3)))))
}
A: NumValue(3) B: NumValue(4) C: NumValue(5) D: NumValue(6)
3
Automated grading
Easy to make (for students)
Easy to guess
Hard to design
Not test of relevant skills

5. Programming Assignments in the Lab
TAs download, build, test, grade manually
Installing software on lab/student machines
Long feedback cycle
Submission, grading, administration workflow
Spreadsheet administration
Uploading grades to Osiris

6. Christmas Break 2011

7. WebLabBrowser
code + test
database
success/all
code + test
success/all
securityusability scalability
An integrated learning management system with support for programming education
assignment
LabBack
execution
engine

8. demonstration

9. Monitoring Student Progress

10. Automatic grading with unit testing
Instructor uses same environment
Program in browser, compile & run on server
WebLab
Use for programming exams
Works everywhere without installation
Scales to large numbers of students
Alignment between teaching & assessment
Objective automated assessment
Automate lab logistics
Rapid (immediate) feedback to students
Reduces tedious work of instructor
Continuous monitoring of student progress
Also: MC, essay questions, grading rubrics

11. Research Agenda
To Do
aka
WebLab Team
Eelco Visser, Vlad Vergu, Elmer van Chastelet, Oscar Castaneda

12. Translating assessment into meaningful grades
Relative weight of tests
Gamification
Programming environment
grow into online IDE
Improving Assessment & Feedback
Reduce effort of creating tests
Generate tests from more declarative specification of correctness
Property-based testing
Testing beyond functional correctness
Non-functional requirements
Recognizing good design
Feedback
How to help students when they are stuck?
Generating hints / solution strategies from assessment
Personalized learning paths

13. Can we apply this to other programming courses?
Algorithms & Data Structures (Java)
Algorithmics (Java)
Embedded Software (C)
(and at courses at TU Darmstadt)
Your course?
Generalizing Automated Assessment & Feedback
Can we apply this to non-programming courses?
Automate correctness testing
Define evaluation function that produces
- a grade (correctness degree)
- feedback (explanation of non-correctness)
Plagiarism
How can we make sure that students really do the work themselves?
My current answer: digital exam with physical presence

14. Monitoring students in a course
We need to record more data
Learning Analytics
Evaluating learning processes
What can we learn from sequence of intermediate
solutions about solution strategies?
Evaluating the effects
Does it work?
Do students learn more? better?
How can we tell?

15. What do you need for
your course in WebLab?
http://weblab.tudelft.nl
WebLab is open for education