Presentation of education paper "Teaching Data-driven Video Processing via Crowdsourced Data Collection" presented at 42nd Annual Conference of the European Association for Computer Graphics (Eurographics’2021).

1. Teaching Data-driven Video Processing
via Crowdsourced Data Collection
Max Reimann1, Ole Wegen1, Sebastian Pasewaldt1,2, Amir Semmo1,2 , Jürgen Döllner1 , Matthias Trapp1
1Hasso Plattner Institute, Faculty of Digital Engineering, University of Potsdam
2Digital Masterpieces GmbH
1

2. Motivation
 Deep-Learning is increasingly also used in CG
 CG-Courses will (need to) adapt
 Classroom projects today either use
 Pretrained models or
 Existing training datasets
 Limitation on scope and quality
 ML is data-driven, easiest way to improve or
enable new applications is to collect more data
requires large, class-specific datasets
GAUGAN [Park et al. 2019]
2
2

3. Motivation
How to acquire new training data in a CG-Course ?
 Annotation by students is infeasible
 Annotation by professionals is too expensive
 Using crowdsourcing marketplaces such as Amazon MTurk is
very flexible but requires overcoming some unique
challenges
□ Designing an intuitive and efficient annotation tool
□ Achieving consistent and high-quality
3
3

4. A Data-Driven CG-Course
Our Goal:
■ Teach hands-on knowledge in data-driven solving of
CG problems in challenging domains such as video
processing
■ Students implement annotation tool, crowdsourced data-
acquisition, model training and CG-effect design in one
seminar project
■ Focus on efficient and effective usage of constrained budget
4
4

5. Course Details
Master’s seminar on image and video processing
6 ECTS ~ 180 hours of study time
Student requirements:
■ Completed CG or image processing lecture
■ Basic understanding of ML methods
Introduction Project Implementation Presentations
• Related work
• Requirements
• Roadmap
• Annotation tool
• Dataset collection
• Model training &
application
• Midterm
presentation
• Endterm
presentation
Seminar components
5
5

6. Exemplary Effect - Contact Visualization
Real-world inspiration Generated effect 6
6

7. Task
Definition
Annotation
Tool
Design
Data
Collection
&
Annotation
Quality
Evaluation
Model
Training
Model
Evaluation
Model
Application
Project Implementation Phases
7
7

8. Student
•Discuss Ideas
•Implement prototype
•Collect & prepare source data
•Improve prototype with feedback
•Model training & application
Supervisor
•Initial task specification
•Give feedback and suggestions
•Lend technical ML-expertise
•Suggest when to move forward to
next stage
•Keep track of progress
Task
Definition
Annotation
Tool
Design
Data
Collection
&
Annotation
Quality
Evaluation
Model
Training
Model
Evaluation
Model
Application
Roles and Responsibilities
8
8

9. Annotation Tool Design
9
9

10. Annotation Tool Design
Tool should balance:
■ Clarity: The task definition and tool usage
have to be made as clear as possible
■ Accuracy: The annotation tool has to
enable annotators to consistently achieve a
high level of accuracy
■ Efficiency: The tool must enable the
annotator to complete a task in the
minimal amount of time possible
Screen from annotation tutorial
10
10

11. Tool Design - First Iteration
11

12. Increasing Accuracy & Efficiency
Accuracy guides
■ Use well-known editing metaphors
■ Zoom-in, keypoint markers
Tool efficiency
■ Present frames subsequently
■ Automatic zoom-in on keypoints
Reducing annotator frustration
■ Retaining efficient workers is beneficial
■ Monetary bonuses can keep best workers
motivated
Zoom-in and keypoints
12

13.  Only 35% of results are correctly annotated
Common mistakes:
Initial Results
13

14. Task
Definition
Annotation
Tool
Design
Data
Collection
&
Annotation
Quality
Evaluation
Model
Training
Model
Evaluation
Model
Application
Tool redesign
Redesign Cycle
14

15. Improved Application
15

16. Quality Assurance
Initial Quiz to assess task
understanding
Annotation task with 10 fixed images
Manual assessment of the results
Issue qualification for workers with
good results
16

17. Results
17

18. Annotation Results
Results: 38.000 well annotated persons in
frames from 760 videos
Cost: 1650$
Observation: Few workers did a lot of the
tasks
■ We rewarded them with an extra bonus
■ We gave them individual feedback for their
work
Chart: Number of tasks each worker
completed
18

19. Annotation Results
Batch HITs Price-per-
HIT
Quiz + keypoint
correction
Qualification task Usable
1 886 $0.18 No No 35%
2 498 $0.18 Yes No 64%
3 1659 $0.30 Yes Yes 82%
4 1565 $0.30 Yes Yes 89%
Data extension batch after additional worker feedback
19

20. Time expenditure
Task
Definition
Annotation
Tool
Design
Data
Collection
&
Annotation
Quality
Evaluation
Model
Training
Model
Evaluation
Model
Application
Tool redesign (2x)
Extension of data (1x)
Architecture
adjustments
1-5 5-10 10-15 15-25
Time, on average, spent per iteration (in h):
Total: ~60h annotation tool, ~40h data collection & evaluation, ~40h model training
20

21. Discussion
21

22. Is data acquisition is a valuable part of a CG/ML curriculum ?
“The quality of the data influences the results so profoundly that it is important to know, how
to acquire such data for training specific tasks”
What were the key learnings?
1) a precise and clear definition of annotation tasks and
2) making the annotation tool as unambiguous as possible, as well as
3) using qualification tasks was important to obtain good training data
Discussion – Student Questionnaire
22

23. Strengths
 Holistic approach and hands-on ML-experience
 Not bound to existing models
 Promotes graduate and undergraduate research
Weaknesses
 Higher technical risk of not achieving satisfactory results
 Relatively high costs
 Arduous process of source data collection
Discussion & Future Work
23

24.  Evaluate in larger context
 Mix with synthetic data generation
Future Work
24