SC4 Hangout - Luigi Selmi, Transport pilot architecture

1. SC4 Pilot Online Hangout
Luigi Selmi - FhG27 June 2016

2. Pilot Scenario
Multisource data collection for the provision of
accurate info-mobility and advanced transport
planning services in Thessaloniki
◎FCD Taxi
◎BT P2P Detectors

3. Present FCD Pipeline
Storage and processing in MSSQL Server

4. Aims of the SC4 Pilot
A new system with Big Data characteristics
◎Volume (Scalability and Fault-tolerance)
◎Velocity (Low latency)
◎Variety (Formats, Unstructured, Structured)
◎Easy Deployment

5. Reference Architecture

6. SC4 Pilot Architecture

7. Semantic Lifting
To improve the quality of the data and make a
better use of it, the semantics of the data will
be made explicit at the ingestion phase using
standard ontologies or creating new ones.

8. SC4 Pilot Components
◎Data Ingestion: Kafka Producer (Flink Jobs)
◎Data Integration: Kafka
◎Data Processing: Flink + Rserve + PostGis
◎Data Storage: Elasticsearch
◎Data Presentation: Kibana
◎Pilot Pipeline

9. Kafka & Flink

10. SC4 Pilot Github Repos
5 Projects on the BDE Github Repository
◎Pilot-sc4-kafka-producer
◎Pilot-sc4-docker-r
◎Pilot-sc4-mapmatcher (for testing)
◎Pilot-sc4-flink-kafka-consumer
◎Pilot-sc4-pipeline (docker-compose)

11. BDE Components
The BDE platform provides
◎Docker images for all the components
◎A tool for integration and initialization based
on Docker Swarm
◎Integrated UI

12. 3 Nodes Configuration

13. Current and Future Work
1. Set up a Docker Swarm for the pilot in 3
nodes (real and VMs) using the dockerized
version of all the components with Init
Daemon and BDE Integrated Dashboard
2. Test the pilot swarm for fault-tolerance and
scalability