Vivek Nagarajan presented on scalable pipelines. His role was to reduce latency and setup infrastructure for scaling pipelines. He described how his ETL pipeline worked and how it was scaled. The pipeline was able to reduce processing time by over 50% and run 30 pipelines concurrently in around 2 minutes each. Possible extensions discussed were running Flink on Yarn and benchmarking with Spark.

1. Scalable Pipelines
Vivek Nagarajan
Insight Data Engineering Consulting Project

3. My Role
• Reduce latency of running a pipeline
• Setup infrastructure for scaling pipelines

4. Pre-Pipeline Stage
<filename, yaml>

5. Pre-Pipeline Stage

6. My ETL Pipeline

7. My ETL Pipeline

8. My ETL Pipeline

9. Scaling Pipeline

10. Scaling Pipeline

11. Scaling Pipeline

12. Scaling Pipeline

13. Demo
Youtube demo:
https://youtu.be/iCMrUqW2n2g

14. Challenges
• Understand existing framework and
infrastructure
• Evolving set of requirements
• Scaling pipelines in flink standalone
cluster mode

15. Performance Stats
• Reduced time taken to process pipeline
by over 50 percent
• Running 30 pipelines concurrently takes
an average of 2 minutes per pipeline

16. Possible extensions
• Running Flink on yarn
• Benchmarking with Spark transformations
• More support for dynamic transformations

17. About Me Vivekanand Ganapathy Nagarajan
Sr. Software Engineer, Rambus Inc.
MS in Computer Science with focus
on Algorithms and Database
University of Southern California
Worked at
Vmware and
Amazon as
Software
Engineer in Test
Passionate about
learning and staying
abreast with technology
I am an avid Coursera
learner having
completed over 20
courses with special
focus on Algorithms

18. Thank You