This is a talk I gave to potential applicants about our pilot of AutoFeedback for term 2 of a first-year Java programming module. AutoFeedback is an automated code feedback platform that tries to lower the barriers to entry as much as possible for submitting code from an IDE, and receiving feedback about it.

1. Boosting individual feedback
with AutoFeedback
April 29th
, 2021

2. Who am I?
● Lecturer in Computer Science at Aston University
● I teach
○ Y1 Java Programming
○ Y2 Team Project
● I do research on software engineering (i.e. making better software):
○ Automating software testing
○ Creating better notations to describe what the system should do
● I am one of the Directors at Beautiful Canoe:
○ Not-for-profit social enterprise at Aston (students develop projects here!)
○ https://beautifulcanoe.com/

3. Motivation for automated code feedback in CS1OOP
● CS1OOP runs through the whole year:
○ Term 1 - intro to programming with Processing (https://processing.org/)
○ Term 2 - object-oriented programming with Java
○ Labs supported by 1 staff member for each 20-30 students
● Hands-on labs each week:
○ Students need to know if they need to fix their code
○ Lecturers need to know what parts of the lab are perceived as difficult
● Manual code feedback limitations:
○ Manually going through hundreds of submissions each week would delay feedback
○ Late feedback may not be useful!
● Automation helps students self-diagnose and staff give faster feedback:
○ The tool can check your code 24/7, and be 100% consistent
○ You can share the results with staff, to identify issues faster

4. Challenges involved
● Submitted code may have bugs - they shouldn't break the server!
● The platform needs to integrate with real tools
○ Professionals use IDEs like Eclipse or IntelliJ to develop their programs
○ Commercial solutions for automated feedback tend to force students to use web-based
code editors, which are not as powerful as the above IDEs
● Submission should be easy for first-time programmers
○ Shouldn't have to learn Git just to send your first program
● The platform needs to support multiple feedback loops:
○ Based on feedback, students will refine their code and re-submit
○ When stuck, students will share their results with staff
○ Based on student feedback, lecturers will refine automated checks and lab worksheets
○ The platform is continuously improved based on the student experience

5. AutoFeedback - our house-grown solution
● We did not find any
commercial option
which ticked all
these boxes
● We made our own:
AutoFeedback
● Piloted in term 2 of
CS1OOP this year
● Processed more
than 18,000
submissions!
AutoFeedback is open source - get the code at https://gitlab.com/autofeedback/

6. AutoFeedback - how does a student submit code?
1. Download the starting code
from our virtual learning
environment (Blackboard)
2. Open the code with your
IDE (e.g. Eclipse or IntelliJ)
3. Go through the lab
worksheet
4. Right-click on a special file
and click on "Submit to
AutoFeedback" (two clicks!)

7. AutoFeedback - how does a student see feedback? (I)
● The submission process will
ZIP your code and send it
to AutoFeedback
○ AF queues submissions: if we
get many at once, you will
just wait a bit longer!
● It also opens a browser tab
that will eventually refresh
with the results
● Keep working on the
feedback and resubmitting!
● Ask staff if something is
unclear!

8. AutoFeedback - how does a student see feedback? (II)
● Each lab has its own
collection of tests, with:
○ Marks for passing the test
○ Feedback if passed
○ Feedback if failed
● Colors:
○ Passed: green
○ Crashed: yellow
○ Failed: green
● AF uses popular testing tools:
○ Maven
○ JUnit 5
○ Mockito
○ JavaParser
○ TestFX

9. AutoFeedback - how is it made?
● AutoFeedback is a system, not just one tool!
● AF has two main parts:
○ The AutoFeedback website - written in PHP using the Laravel web framework
■ Runs 24/7 inside a server in Aston, with automated backups and system upgrades
○ The AutoFeedback Maven plugin - written in Java
■ Runs from the IDE in your computer - submits your code and opens the browser tab

10. AutoFeedback - how is the website made? (I)
● Remember the risks around letting buggy code crash our server?
● To protect against that, we use "containerization" with Docker:
○ Each container is effectively running as if they were on a separate computer
○ We can impose CPU/disk/memory limits on containers - each submission gets:
■ Half a CPU core, for up to 10 minutes
■ 100MB of disk space - gets erased automatically after marking
■ 512MB of memory
● We use containers for all the other parts of the AF website, too
○ Containers ensure the website is set up exactly the same everywhere
○ No more "but it works on my machine!" :-)

11. AutoFeedback - how is the website made? (II)
● Our Docker containers:
○ nginx is the web server - it receives
HTTP(S) connections and serves static
files (CSS, JS, images) directly.
○ "app" is the application server - it renders
web pages and schedules jobs to process
student code into a queue.
○ The "default-worker"s process non-Java
background jobs coming from "app" (e.g.
marking or importing Blackboard users).
○ The "java-worker"s run student code
under CPU/RAM/disk limits.
○ We use MariaDB for storing data, and
Redis for storing push notifications and
the job queue.
○ Laravel Echo delivers push notifications.

12. AutoFeedback - how is the website kept up to date?
● We use the Git version control system to track all our changes to AF
● The AF code is hosted in Gitlab: all changes undergo automated tests
○ This is known as Continuous Integration (CI)
● If the code passes all tests, it is automatically deployed in the server:
○ This is known as Continuous Delivery (CD)
○ CD prevents breaking the server due to a mistake doing some manual upgrade step

13. AutoFeedback - what is next?
● Supporting other languages besides Java:
○ Planned redesign around the Kubernetes orchestration platform
○ Could support anything that produces a JUnit XML file (there
are testing tools like that for PHP, JavaScript, Python…)
● Improving the feedback:
○ Allowing students to ask questions in AutoFeedback if it's not clear enough
○ Integrating videos on top of text for feedback
● Showing lecturers the most common mistakes students make
● Supporting "micro-assessments" with code written from the website
○ Good for first-time programmers doing very short programs
○ May be useful for the first few weeks!

14. Thank you!
@antoniogado
a.garcia-dominguez@aston.ac.uk