Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

SC1 Workshop 2 Technical overview


Published on

Technical overview of the BDE platform as presented by Aad Versteden at the SC1.2 workshop in Brussels

Published in: Technology
  • Login to see the comments

  • Be the first to like this

SC1 Workshop 2 Technical overview

  1. 1. SC1 Health Workshop Technical overview4 Oct 2016
  2. 2. Platform goals ◎ Low total cost of ownership ◎ Simple to get started with Big Data ◎ Cater for widely varying use cases ◎ Embrace emerging Big Data technologies ◎ Simple integration with custom components
  3. 3. Key actors
  4. 4. Big Data is ◎ Volume o Quantity of data ◎ Velocity o Speed at which data is provided ◎ Variety o Different formats/models in which data is provided ◎ Veracity o Accuracy/truthfulness of the data Why did we need all this?
  5. 5. Platform architecture
  6. 6. Platform architecture
  7. 7. Platform architecture
  8. 8. Semantic Big Data ongoing research! ◎ Semantic Data Lake o from data swamp to data lake o query contents in the data lake ◎ SANSA stack o Big Data analytics on semantic graph
  9. 9. Support layer ◎ Swarm UI o Launch, install and manage pipelines ◎ Pipeline daemon & monitor o Determine order in which steps are executed o eg: Upload files before running computations ◎ Integrator UI o Present dashboards in a unified interface
  10. 10. Platform architecture
  11. 11. Key actors
  12. 12. Platform installation
  13. 13. Platform installation ◎ Manual installation guide ◎Using Docker Machine o On local machine (VirtualBox) o In the cloud (AWS, DigitalOcean, Azure) o Bare metal ◎ Screencast
  14. 14. Platform development
  15. 15. ◎ High level picture o docker-compose.yml describes pipeline topology ◎ Common components o extend template image with your code ◎ New components o build a Docker image for your component o this is your own little Virtual Machine for your component ◎ Sharing o publish topology as git repository o publish new components on docker hub Platform development
  16. 16. Platform development
  17. 17. Deployment
  18. 18. Swarm UI
  19. 19. Swarm UI
  20. 20. Deployment
  21. 21. Swarm UI
  22. 22. Swarm UI
  23. 23. Integrator UI
  24. 24. Workflow UI
  25. 25. More monitoring This topic is ongoing ◎Custom User Interfaces ◎System output logs ◎Monitor network wire format (and visualise)? ◎Monitor node load (and autoscheduling)?
  26. 26. Concluding remarks ◎Used in practice ◎Easy to get started ◎Improving as we speak
  27. 27. You can talk to us! ◎Aad Versteden ◎