General deployment solution for big data platforms, most of the platforms are master-agent architecture, ex: Cloudera, Mesos, ...

1. Big Data Platform
Deployment
David Ko

2. Goals
General deployment solution for big data platforms, most of the platforms are master-
agent architecture, ex: Cloudera, Mesos, ...
2

3. Elements to Achieve
Roles
Flow
Status
Remote Communication
Self Healing
Deployment Implementation (exclusive)
3

4. Roles, Flow, Status
Coordinator
Executor
Reporter
Action
Message
Deploy Executor
(ex: Cloudera)
Deploy
Coordinator
Deploy Status
Reporter
Deploy
Action
Deploy
Action
Deploy
Action
4

5. Remote Communication
No Cluster here, just base on node list, persistent status, seamless REST API and
periodical thread
Deploy Executor
(Cloudera)
Deploy
Coordinator
Deploy Status
Reporter
Deploy
Action
Deploy
Action
Deploy
Action
Deploy Executor
(Cloudera)
Deploy
Coordinator
Deploy Status
Reporter
Deploy
Action
Deploy
Action
Deploy
Action
REST API
5

6. Self Healing
Do status restore check at service start or action error
Deploy Executor
(Cloudera)
Deploy
Coordinator
Deploy Status
Reporter
Deploy
Action
Deploy
Action
Deploy
Action
1
2
6

7. Deployment Implementation
Implement specific actions to do the big data platform deployment
7

8. Architecture
Deploy Executor
(Cloudera)
Web API
Deploy
Coordinator
Deploy Status
Reporter
Deploy API
(controllers)
Deploy API
(controllers)
Deploy
Coordinator
Master Agent
Deploy
Action
Deploy
Action
Deploy
Action
Deploy Executor
(Cloudera)
Deploy Status
Reporter
Deploy
Action
Deploy
Action
Deploy
Action
8

9. Thank You.
9