Insight Data Engineering project to build scalable pipelines.

1. Scalable Pipelines
Vivek Nagarajan
Insight Data Engineering Consulting Project

3. My Role
• Reduce latency of running a pipeline
• Setup infrastructure for scaling pipelines

4. Pre-Pipeline Stage
<filename, yaml>

5. Pre-Pipeline Stage

6. My ETL Pipeline

7. My ETL Pipeline

8. My ETL Pipeline

9. Scaling Pipeline

10. Scaling Pipeline

11. Scaling Pipeline

12. Scaling Pipeline

13. Demo
Youtube demo:
https://youtu.be/iCMrUqW2n2g

14. Challenges
• Understand existing framework and
infrastructure
• Evolving set of requirements
• Scaling pipelines in flink standalone
cluster mode

15. Performance Stats
• Reduced time taken to process pipeline
by over 50 percent
• Running 30 pipelines concurrently takes
an average of 2 minutes per pipeline

16. Possible extensions
• Running Flink on yarn
• Benchmarking with Spark transformations
• More support for dynamic transformations

17. About Me Vivekanand Ganapathy Nagarajan
Sr. Software Engineer, Rambus Inc.
MS in Computer Science with focus
on Algorithms and Database
University of Southern California
Worked at
Vmware and
Amazon as
Software
Engineer in Test
Passionate about
learning and staying
abreast with technology
I am an avid Coursera
learner having
completed over 20
courses with special
focus on Algorithms

18. Thank You