Technical requirements for publishing live open data in Flanders, Belgium.

1. A Preliminary Open Data
Publishing Strategy for Live
Data in Flanders
Julián Andrés Rojas Meléndez
Brecht Van de Vyvere
Pieter Colpaert
Ruben Verborgh
Arne Gevaert
Ruben Taelman

2. Public Authority

3. Public Authority
Air Quality
Noise Levels
Street Occupancy
Parking Availability
Temperature
Traffic Lights
Council Decisions
Demographics
Tax Utilization

4. Smart Flanders program*
*https://smart.flanders.be/

5. How to publish live datasets as
Open Data for maximum reuse?

6. How to publish live datasets as
Open Data for maximum reuse?

7. All datasets that can be modeled using
a ledger-like model

8. Public Authority
Air Quality
Noise Levels
Street Occupancy
Parking Availability
Temperature
Traffic Lights
Council Decisions
Demographics
Tax Utilization

9. Public Service
Parking Availability

10. https://linked.open.gent/parking

11. https://linked.open.gent/parking
https://linked.open.gent/parking?from=2018-03-01&to=2018-03-30

12. https://linked.open.gent/parking
https://linked.open.gent/parking?from=2018-03-01&to=2018-03-30
https://linked.open.gent/parking?from=2018-03-01&to=2018-03-30&filter=electric

13. https://linked.open.gent/parking
https://linked.open.gent/parking?from=2018-03-01&to=2018-03-30
https://linked.open.gent/parking?from=2018-03-01&to=2018-03-30&filter=electric

14. Requirements for live open data on the Web
Proof of concept: Live parking availability data
Consuming live data: Polling vs. Pubsub

15. Requirements for live open data on the Web
Proof of concept: Live parking availability data
Consuming live data: Polling vs. Pubsub

16. Open Data for maximum reuse?
Reduce
adoption costs
Legal
Technical
Syntactic
Semantic
Querying
Open Data licenses
(https://opendefinition.org/ )
HTTP (?)
RDF
LOD vocabularies and
domain ontologies
RSP (C-SPARQL, CQELS,
Strider, etc)

17. Public Authority
{
Large heterogeneity of questions
Highly available interface
Low latencies for data responses
Discoverable information resources
Automatic discovery of legal
constraints
Cross origin resource sharing
Access to historic data

18. Parallels with On a Web of Data Streams*
We share reusability requirements
metadata, standards, non-functional requirements
We focus on data publishers while they work on
the entire streaming ecosystem
Stream query processors, reasoners, visualizers
We don’t prioritize pushing techniques
*D. Dell’Aglio, D. Le Phuoc, A. Le-Tuan, M. I. Ali, and J.-P. Calbimonte. On a web of data streams. 2017.

19. Requirements for live open data on the Web
Proof of concept: Live parking availability data
Consuming live data: Polling vs. Pubsub

20. What does the data look like?
S = (G1
, t1
), (G2
, t2
), …, (Gn
, tn
)

21. Data pages for historic and live data
time
NowHistoric pageHistoric page
…

22. Visualize parking availability live and historic data

23. Polling limitations when dealing with live data
Sub-second updates are not cacheable
Handling asynchronous updates

24. Requirements for live data on the Web
Proof of concept: Live parking availability data
Consuming live data: Polling vs. Pubsub

25. Hypothesis
Clients
CPU / Response Time
Polling
Pubsub
Polling-based strategies will outperform Pubsub-based ones after a
certain number of clients thanks to caching capabilities for streams
with synchronous updates and T >= 2s

26. No clear winner on CPU usage

27. Pubsub latencies are lower
Latencies density distribution - Data updates each 2s
Pubsub
(websockets)
Polling (http)

28. Requirements for live
open data on the web
Proof of concept: Live
parking availability data
Consuming live data:
Polling vs. Pubsub

29. Discussion
Set of recommendations to publish live datasets on the
web based on its properties
Multidimensional interfaces for a distributed ecosystem of
query optimized live data
Should we consider sub-second caching for HTTP?
Julián Andrés Rojas Meléndez
julianandres.rojasmelendez@ugent.be
@julianr1987