The document provides an overview of the SC1 Health Workshop technical platform. The platform goals are low cost of ownership, ease of use with big data, flexibility for different use cases, embracing emerging big data technologies, and simple integration. The platform architecture uses Docker containers and Compose files to define the pipeline topology. Components are developed as Docker images and the platform can be installed manually or using Docker Machine on various environments.

1. SC1 Health Workshop
Technical overview4 Oct 2016

2. Platform goals
◎ Low total cost of ownership
◎ Simple to get started with Big Data
◎ Cater for widely varying use cases
◎ Embrace emerging Big Data technologies
◎ Simple integration with custom components

3. Key actors

4. Big Data is
◎ Volume
o Quantity of data
◎ Velocity
o Speed at which data is provided
◎ Variety
o Different formats/models in which data is provided
◎ Veracity
o Accuracy/truthfulness of the data
Why did we need all this?

5. Platform architecture

6. Platform architecture

7. Platform architecture

8. Semantic Big Data
ongoing research!
◎ Semantic Data Lake
o from data swamp to data lake
o query contents in the data lake
◎ SANSA stack
o Big Data analytics on semantic graph

9. Support layer
◎ Swarm UI
o Launch, install and manage pipelines
◎ Pipeline daemon & monitor
o Determine order in which steps are executed
o eg: Upload files before running computations
◎ Integrator UI
o Present dashboards in a unified interface

10. Platform architecture

11. Key actors

12. Platform installation

13. Platform installation
◎ Manual installation guide
◎Using Docker Machine
o On local machine (VirtualBox)
o In the cloud (AWS, DigitalOcean, Azure)
o Bare metal
◎ Screencast

14. Platform development

15. ◎ High level picture
o docker-compose.yml describes pipeline topology
◎ Common components
o extend template image with your code
◎ New components
o build a Docker image for your component
o this is your own little Virtual Machine for your component
◎ Sharing
o publish topology as git repository
o publish new components on docker hub
Platform development

16. Platform development

17. Deployment

18. Swarm UI

19. Swarm UI

20. Deployment

21. Swarm UI

22. Swarm UI

23. Integrator UI

24. Workflow UI

25. More monitoring
This topic is ongoing
◎Custom User Interfaces
◎System output logs
◎Monitor network wire format (and visualise)?
◎Monitor node load (and autoscheduling)?

26. Concluding remarks
◎Used in practice
◎Easy to get started
◎Improving as we speak

27. You can talk to us!
◎Aad Versteden
aad.versteden@tenforce.com
◎https://github.com/big-data-europe