Overview of the general BDE platform architecture in the Webinar (1st in a series of 8) on 02.03.2016

1. 1ST TECHNICAL WEBINAR
BDE Platform Architecture02.03.2016

2. Targets of the BDE platform
1-mars-16www.big-data-europe.eu
 Easy to use

3. 1-mars-16
 Flexible

4. Applications running on the platform
1-mars-16www.big-data-europe.eu
 Pipeline
o Communicating components
to solve a specific problem
 Example
o HDFS nodes
o Spark master/workers
o Spark map-reduce algorithm

5. Platform architecture
1-mars-16www.big-data-europe.eu

6. Platform architecture
1-mars-16www.big-data-europe.eu

7. Platform architecture
1-mars-16www.big-data-europe.eu

8. 1-mars-16www.big-data-europe.eu

9. Installation
 Manual installation
o https://github.com/big-data-europe/README/wiki/Base-platform
 Chef recipes
o Temporarily outdated
o https://github.com/big-data-europe/chef-base-platform-cookbook
1-mars-16www.big-data-europe.eu

10. 1-mars-16www.big-data-europe.eu

11. Docker in a nutshell
 Docker
o Build, ship and run any application on any platform
 Docker containers
o =~ lightweight virtual machine
 Docker image
o Template of a Docker container
1-mars-16www.big-data-europe.eu

12. Developing a component
1-mars-16www.big-data-europe.eu
 Base Docker images
provided by BDE
 Extend image with
custom algorithm/data
with minimal effort

13. Developing a pipeline
1-mars-16www.big-data-europe.eu
 Pipeline = collection
of communicating
components
 Describe pipeline
in 1 file using
Docker Compose

14. Supported components
- https://github.com/big-data-
europe/README/wiki/Components
- https://hub.docker.com/u/bde2020
1-mars-16www.big-data-europe.eu
GitHub
DockerHub

15. 1-mars-16www.big-data-europe.eu

16. 1-mars-16
docker-compose up
Custom GUI is being built

17. Docker Swarm
 Resources are managed by Swarm
 Integrates with Docker Compose
 Automatic download/install/run
1-mars-16www.big-data-europe.eu

18. 1-mars-16www.big-data-europe.eu

19. Future work: Monitoring
1-mars-16www.big-data-europe.eu

20. 1-mars-16
THANK YOU