This document discusses using Mesos and Spark together to build a fault tolerant data pipeline. It outlines the architecture of connecting data sources to the pipeline, implementing multiple pipelines, ensuring high availability through Mesos, and performance monitoring. Learnings and a conclusion are also presented.

2. Use Case

3. Roadblocks

4. Our Saviors

5. • Spark => In-memory Execution Engine
• Mesos => Resource Manager
• Mesos + Spark => ??

6. Architecture

7. Connecting Sources to Pipeline
❖
➢
➢
❖
➢

8. Multiple Pipeline Implementation

10. Fault Tolerance Pipeline

11. MESOS: High Availability

12. Performance & Monitoring

13. Learnings

14. Conclusion