Design of big data framework for building complex data pipelines

1. Data Pipeline
Framework
Dmitry Shirokov

2. Core Features
Fast-paced pipeline
development
Scalability Flexibility
Secure Mature
Code generation of
pipeline based on
patterns
Supportable and
Maintainable
Schema Evaluation
Support

3. Framework design: core Interfaces

4. Example of code generation pipeline component based on interfaces

5. Example of complex pipeline

6. Example of integration by using framework