The document provides an overview of the SC1 Health Workshop technical platform. The platform goals are low cost of ownership, ease of use with big data, flexibility for different use cases, embracing emerging big data technologies, and simple integration. The platform architecture uses Docker containers and Compose files to define the pipeline topology. Components are developed as Docker images and the platform can be installed manually or using Docker Machine on various environments.
2. Platform goals
◎ Low total cost of ownership
◎ Simple to get started with Big Data
◎ Cater for widely varying use cases
◎ Embrace emerging Big Data technologies
◎ Simple integration with custom components
4. Big Data is
◎ Volume
o Quantity of data
◎ Velocity
o Speed at which data is provided
◎ Variety
o Different formats/models in which data is provided
◎ Veracity
o Accuracy/truthfulness of the data
Why did we need all this?
8. Semantic Big Data
ongoing research!
◎ Semantic Data Lake
o from data swamp to data lake
o query contents in the data lake
◎ SANSA stack
o Big Data analytics on semantic graph
9. Support layer
◎ Swarm UI
o Launch, install and manage pipelines
◎ Pipeline daemon & monitor
o Determine order in which steps are executed
o eg: Upload files before running computations
◎ Integrator UI
o Present dashboards in a unified interface
13. Platform installation
◎ Manual installation guide
◎Using Docker Machine
o On local machine (VirtualBox)
o In the cloud (AWS, DigitalOcean, Azure)
o Bare metal
◎ Screencast
15. ◎ High level picture
o docker-compose.yml describes pipeline topology
◎ Common components
o extend template image with your code
◎ New components
o build a Docker image for your component
o this is your own little Virtual Machine for your component
◎ Sharing
o publish topology as git repository
o publish new components on docker hub
Platform development
25. More monitoring
This topic is ongoing
◎Custom User Interfaces
◎System output logs
◎Monitor network wire format (and visualise)?
◎Monitor node load (and autoscheduling)?