"The DEBS Grand Challenge 2018" as presented in the 12th ACM International Conference on Distributed and Event-Based Systems (DEBS 2017), 25 - 29 June, 2017 held in Hammilton, New Zeland This work was supported by grants from the EU H2020 Framework Programme provided for the project HOBBIT (GA no. 688227).

1. THE DEBS 2018 Grand
Challenge
Vincenzo Gulisano, Zbigniew Jerzak, Pavel Smirnov,
Martin Strohbach, Holger Ziekow, Dimitris Zissis

2. Agenda
•This challenge problem 2018
•Evaluation process
•How this finale works

3. The DEBS Grand Challenge
 Started in 2011 (8th Grand Challenge in 2018)
 Provides a common ground and uniform evaluation
criteria for a competition aimed at both research
and industrial event-based systems.

4. New Endeavors in 2018
• Predictions about ship
routes based on AIS data
(movement etc.)
• Open solution: no
predefined algorithm
• Machine learning
combined with stream
processing
• Performance Award not
only about speed but also
prediction quality

5. The Problem

6. The Problem:
Predicting Arrival of Ships
• It is not always evident where a ship is going and
when it will be arriving
• AIS data include information about destination and
arrival time
• However
• It is entered by the crew so prone to human error and
inaccuracies (e.g. naming conventions)
• Often does not include intermediate stops on the way to
the destination

7. The Data
• Data provided by
MarineTraffic hosted by
the BIG Data Ocean, EU
Horizon 2020 project
• Data contains AIS
records (location,
movement etc.) over
approx. 3 month

8. The Data Details
• SHIP_ID is the anonymized id of the ship
• SHIPTYPE is defined according to the reference
• SPEED is measured in knots (divide value by 10)
• LON is the longitude of the current ship position
• LAT is the latitude of the current ship position
• COURSE is the direction in which ship moves
(see: https://en.wikipedia.org/wiki/Course_(navigation))
• HEADING (see: https://en.wikipedia.org/wiki/Course_(navigation))
• TIMESTAMP is the time at which the message was sent (UTC)
• DEPARTURE_PORT_NAME is the name of the last port visited by the vessel
• REPORTED_DRAUGHT of a ship's hull is the vertical distance between the
waterline and the bottom of the hull
(keel) https://en.wikipedia.org/wiki/Draft_(hull)
The AIS is a collaborative, self-reporting system
for marine vessels to broadcast their

9. Problem Details
• We consider a list of ports
that ships leave and enter
• Leaving and entering is
defined trough a zone
around the port center
• Predictions are to be made
while ships move between
ports
• Where is the ship going?
• When will the ship arrive?
• Training data is provided to
build the solution

10. Query 1: Predicting Destination of
Vessels
• For each incoming tuple you must provide a
prediction for the destination
• A good solution sticks to the correct prediction as
early as possible
consecutive correct predictions
prior to arrival
𝐴 =
𝑘=0
𝑁𝑡𝑟𝑖𝑝𝑠 𝑙𝑒𝑛(𝑙𝑎𝑠𝑡 𝑐𝑜𝑟𝑟𝑒𝑐𝑡 𝑠𝑒𝑞𝑢𝑒𝑛𝑐𝑒)
𝑙𝑒𝑛(𝑎𝑙𝑙 𝑡𝑢𝑝𝑙𝑒𝑠 𝑜𝑓 𝑎 𝑡𝑟𝑖𝑝)
𝑁𝑡𝑟𝑖𝑝𝑠

11. Query 2: Predicting Arrival Times
of Vessels
• For each incoming tuple you must predict when
this ship will arrive at a listed destination
• A good solution has a small average error
𝐴 = 𝑖=0
𝑁𝑡𝑢𝑝𝑙𝑒𝑠
𝑎𝑏𝑠(𝑒𝑟𝑟𝑜𝑟 𝑝𝑒𝑟 𝑡𝑢𝑝𝑙𝑒)
𝑁𝑢𝑝𝑙𝑒𝑠

12. Sample Tracks

13. The overall Score
• For each query solutions are ranked by
• (A) prediction quality
• (B) by speed (total runtime)
• Prediction quality is weighted 75% and speed 25%
 avoiding fast meaningless solutions
• Final rank is the average of Query 1 and 2

14. 25.10.2018 14
The HOBBIT platform
23
.
1
5
6Customer
Requires ranking of alternative
solutions by some KPI
Solution provider (vendor)
(e.g. DB, Streaming Platforms, ML
frameworks, etc…)
Customer
Requires ranking of alternative
solutions by some KPI
Benchmark customer
Requires ranking of alternative
solutions by some KPI
Provides:
1. Manual benchmark executions
2. Bulk executions (challenge)
3. GUI, Leaderboards
Advantages:
1. Batch/streaming benchmarks
2. Docker virtualization
3. RDF-enabled
Submit
benchmarks
Submit
systems
The HOBBIT platform
(online or local instance)

15. 25.10.2018 15
The HOBBIT platform. Architecture
(Simplified for Grand Challenge)
Benchmark components Platform components
1
2
3.1
3.2
4
5
6
The online platform:
http://master.project-hobbit.eu/
Cluster: 6 nodes, each is
2×64 bit Intel Xeon E5-2630v3
(8-Cores, 2.4 GHz, HT, 20MB
Cache, each proc.), 256 GB RAM,
1Gb Ethernet
Nodes (benchmark/system): 3/3
https://github.com/hobbit-project/platform/wiki/Overview
7
http://github.com/hobbit-project/platform
System
components

16. 25.10.2018 16
The DEBS GC 2018 Benchmark
Sequential loop for each ship,
but parallel loops for ships
(> 500 ships)
Benchmark
Controller Task Generator
Evaluation storage /
Evaluation module
Benchmarked
System
Acknowledgement (encrypted task ID)
Deploys
Data tuple with task ID
Expected responses
Result with task ID
Result model (KPIs)
Deploys
Pairs evaluation
Start signal
Termination signal
System finished signal
Generation finished signal
http://github.com/hobbit-project/sml-benchmark-v2

17. Dataset Statistics
• 2 Input sets (a and b) available for GC:
○ about 500.000 tuples each
○ timespan: approx. 3 month
○ about 500 ships in each set (100 overlapping) about 900 in total
• Initial released data set
○ First release: data set a: 80% released, 5% online training with
HOBBIT, kept 15% for final testing
• Second release (after you have running system):
○ data set b: 70% released, 30% kept for final testing
• Final Benchmark
○ 15% from set a + 30% from set be for final benchmark

18. 25.10.2018 18
Challenges

19. 25.10.2018 19
Training phase

20. 2018 Statistics
17 Teams
registered in
EasyChair
9 Teams passed the
checks for inclusion
in DEBS
4 finalists
in this session
(5 others as posters)
(paper review)
Top 2 determined
by Bechmark
Top 2 determined
by reviews

21. Scores
Winner
2 of 8 working systems did not finish processing within
time limit

22. Awards
• $1000 “Grand Challenge Award”
• for the best performing submission
Make your
vote count!
Audience Awards (Voted for by you!)
• Most interesting/appealing solution
• Voting boxes and valleys at the exits
Awards will be announced during the banquett

23. ACKNOWLEDGMENTS
This year’s Grand Challenge is co-organized by the HOBBIT project
(https://project-hobbit.eu/) represented by AGT International (http:
//www.agtinternational.com/). We would like to thank MarineTraffc
(https://marinetrafc.com) and the BigDataOcean project (EU
Horizon 2020, grant agreement number 732310) for providing the
dataset for this challenge. The automated evaluation platform was
provided by the HOBBIT project, while the data will be hosted
on the BigDataOcean project. The HOBBIT project has received
funding from the European Union’s H2020 research and innovation action
program under grant agreement number 688227, while
the BigDataOcean project has received funding from the European
Union’s H2020 research and innovation action program under grant
agreement number 732310.