The SERF project aims to create a structured exercise repository with automated feedback to help students practice programming. It will contain a variety of exercises from multiple institutions, tagged with concepts in a knowledge graph to help students find relevant practice. In a test run, exercises from TU/e and OU courses were added. Students can search for exercises by concept, submit solutions, and get immediate feedback on correctness and style. The project seeks more partner institutions to contribute additional exercises and help evaluate the tool.

1. SERF
Structured Exercise Repository with Feedback
TU/e: Kees Huizing, Ruurd Kuiper and Erik Scheffers
OU: Lex Bijlsma, Arjan Kok, Harrie Passier, Stefano Schivo, Tanja Vos
SURF Onderwijsdagen, 5 november, Den Bosch

2. Agenda
• Introduction and overview
• First and second run
• Demo

3. Why this project?
Situation
- Programming is a complex activity
- Practicing with solving problems is essential
- Popular approaches:
1. Exercises in books (solutions at the back)
2. Tutorials (lecturers guide students)
Problems with these approaches
1. What if the student’s solution deviates? Is my solution correct too?
2. What in cases of huge numbers of students? Is labour-intensive/impracticable
Both. Developing exercises is very labour-intensive
Opportunities
- OO programming languages are stable
- There exist a lot of exercises (scattered over several institutions)
- There are thousands of students / tens of institutions

4. Project goals
Higher level
- Students can practice where/when they like and receive immediate feedback
- Lecturers have insight in students’ progress and quality of exercises
- More (exercises) for less (labor hours)
Lower level
- Proper set of exercises of high quality
- Open: e.g., using and adding of exercises, for several institutions
- Student selects exercises depending on topic and level of knowledge
- Independent of didactic style, e.g. objects first versus objects late
- Immediate and automatic feedback per exercise on correctness and style
- Tool set: web based and stand alone
- Java, HBO/WO programming courses

5. Products
(Fore-) knowledge graph
- Labelling exercises with concepts (lecturer)
- Searching and selecting exercises (students)
Databases with exercises and statistics
Feedback engine
Web / standalone application
- Student side
- Looking for and selecting of exercises
- Submitting solutions
- Receiving feedback
- Lecturer side
- Adding / labelling new exercises with feedback
- Observing students’ progress and quality of exercises
Exercises
Feedback engine
Lecturer side Student side
Statistics
Knowledge graph

6. Project plan
WP 1 – Project management
WP 2 – Foreknowledge graph and Database with exercises
WP 3 – Application and integrated web environment
WP 4 – Evaluation
WP 5 – Application
WP 6 – Knowledge and product dissemination
WP 7 – Preserving and anchoring results
This presentation
This presentation
This presentation

7. Knowledge graph – why?
Students should be able to select exercises from a database corresponding to their knowledge
level (existent/aimed for)
This should be course independent, so
- there is no general order of knowledge-increase
- Example: Object-First versus Object-Late, …, etc.
- not even general “clustered” knowledge levels that exercises can be ordered by
- Example: Methods complete (static, instance methods, parameters, return value, …)
versus First static methods only and later instance methods
So exercises are tagged, tags indicate exercise-relevant knowledge
Problem: lots of tags needed to indicate all bits of such knowledge

8. Knowledge graph – nodes and arrows between nodes
A knowledge graph of general dependencies between bits of knowledge can reduce the
number of tags required for selecting an exercise
– To improve searching and searchability
Nodes (used as tags) – two kinds
Syntax nodes – Syntax elements in exercise (solution program)
Examples: boolean, switch, …
Concept nodes – Semantic concepts
Examples: overloading, dynamic binding, …

9. Knowledge graph – nodes and arrows between nodes
Tags are organized in a graph to better understand their meaning and to simplify the tagging of
exercises – three kinds of arrows
needs A  B knowledge of A needs knowledge of B
casting  types, encapsulation  access
transitive
uses A - -> B knowledge of A uses some knowledge of B
public static final main
not transitive
is-a A B subtyping-like
char is-a primitive type is-a type
transitive

10. Knowledge graph
imperative part

11. Knowledge graph
class part

12. Knowledge graph
inheritance part

13. Knowledge graph – tagging exercises
Tags can be needed or used for an exercise!
Needed: Central issue in exercise
Uses: `also needed’ hoeft niet begrepen te worden.
- Lecturer adds needs and uses to an exercise
- Student chooses needs (positive selection): For-loop
- System shows all exercises tagged with these
- Student looks at uses (possibly negative selection): Collections
- Student chooses negative tags (possibly negative selection): Collections
- System shows reduced list
- Student submits solution and receives feedback

14. First test run
Tu/e Bachelor Java programming course – now running
- inclusion of course exercises in the repository, test whether this is student-assistant proof
(tagging, creating feedback)
- use by students
- some logged student feedback, some further logging for evaluation
OU Java programming course – now running
NB The courses are quite different in style: objects late (TU/e) middle (OU)

15. Second test run
Partners? Mail us!
c.huizing@tue.nl
r.kuiper@tue.nl
harrie.passier@ou.nl
Wanneer: Februari
Functionaliteit via webinterface

16. Collaboration
Issues:
- Quantity of exercises: through submission by partners/users
- One central database
- Quality assurance:
template/guidelines for submitting/tagging exercises (+ feedback)
unpublished/reviewed/published cycle
moderator/maintenance to be discussed

17. Repository Tool
Database and server
Server run at TU/e currently
Open, anonymous for students
Lecturer interface (registered users)
Student interface

18. Student – exercise search
Student chooses needs (positive)
concepts she masters or is mastering
Gets all exercises tagged with these
may look at the uses tags
Too many hits? Chooses negative tags
Gets reduced list
Chooses an exercise

19. Student – exercising
Presented with template
Enters program
directly or prepared in own IDE
Submits
Gets feedback
based on unit tests or other tests
Corrects, resubmits, etc.

20. Teacher
Edits knowledge graph (adds or removes edges)
Reviews exercises
Adds exercises
provides description
provides template (optional)
adds tags
provides test script