BDE SC4 Hangout - Hajira Jabeen, general architecture

1. BDE Platform Architecture
SC 4 Webinar27 June 2016

2. Agenda
◎ Platform architecture
◎User interfaces

3. Platform architecture

4. Platform architecture

5. Platform architecture

6. Platform architecture

7. Platform installation
◎ Manual installation guide
◎Using Docker Machine
o On local machine (VirtualBox)
o In the cloud (AWS, DigitalOcean, Azure)
o Bare metal

8. Platform components
Base Docker images:
◎Serves as a template for a (Big Data) technology
◎Easily extendable
with custom algorithm/data
◎Automating as much as possible
◎May be opinionated

9. Platform components
◎Published components
o Image repositories on GitHub
o Automated builds on Docker Hub
o Documentation on BDE Wiki
o docker-compose snippet in README

10. Developing a Pipeline

11. Big Data pipelines
◎Run pipeline:
$ docker-compose up
◎Orchestrator required for initialization process
o Components may depend on each other
o Components may require manual intervention

12. Platform architecture

13. User Interfaces

14. User interfaces
◎Workflow UIs
o Workflow Builder
❖ Build a workflow to configure the init daemon service
o Workflow Monitor
❖ Monitor a pipeline's initialization process
◎Swarm UI
o Deploy a pipeline on the platform
◎Integration of existing UIs
o BDE CSS based on Materialize CSS
o Integrator UI: present GUIs in a unified interface

17. ◎UI to follow-up status of the pipeline
o Status of each component
o Manually finish a step in the pipeline
◎Integrated as component in
the application's docker-compose.yml
Pipeline monitor application

18. Thank you