The document discusses designing an ETL (extract, transform, load) framework. It outlines goals like implementing routines quickly with limited resources, reliability, flexibility to modify rules, and visibility into status. Architecture considerations include server organization, database access methods, and how SSIS packages will run. Features include package templates, multiple run options, error handling, logging, and notifications. SQL and SSIS components are described like framework tables, stored procedures, package naming, and compartmentalizing work. Web components allow operations, threshold maintenance, notifications, and reporting. The overall goal is a reusable, standardized, and visible ETL process.

1. Designing and Implementing
an ETL Framework
Mike Litchfield
Assistant Vice President – IT
Architect & Sr. Developer
Income Research & Management
mlitchfield@incomeresearch.com

2. Agenda
• Why build an ETL framework ?
• Architecture Considerations
• Reference Architecture
• What features to include ?
• Walk through sql and ssis components
• Walk through web-based components

3. Why Build a Framework ?
• Definition
• Reusability Efficiency Time-To-Market
• Code Standardization Ease-Of-Support Minimized Troubleshooting
• Adherance to best practices
• Functional Flexibility and Scalability
• Accessability to Data Flow Status
• End User Notification

4. IRM Goals
• Implement many routines quickly, with limited developer resources
• Reliability and Accuracy
• Ability to introduce modify remove transformation rules
• Ability to maintain and apply logical business rules on data
• Support for scheduled and user-initiated package execution
• Visibility into ETL Status
• Error Notification and Ability to Debug Quickly
• End User Confirmation
• Support for logical dependencies between Sql Agent steps and SSIS
packages
• Package instance concurrency handling
• Historical persistence of data, errors, overrides, thresholds

6. Architecture Considerations
• Organization of database servers and databases
• Sql Server 32-bit vs 64-bit
• Database access methodology
• How ssis packages will be invoked
• Checkpoint restart database transactions
• Compartmentalizing ssis functionality
• Linked Server
• Excel vs tables

7. What features to Include ?
• Ssis package templates – parent and child
• Multiple ways to invoke the package
• Package instance concurrency handling
• Run date Override date handling
• Tracking package progress
• Error handling and notification
• Variables definition
• Standardized flow
– Concurrency Check
– Registration
– Extract
– Transform re-process corrections
– Rule execution
– Final destination posting
– Historical archiving
– Notification

9. Sql Components
Framework Tables
• Etl_pkg
• Etl_pkg_auth
• Etl_pkg_req
• Etl_control
• ETL_batch_override_dates
• ETL_data_supplier_vendor
• Ssis_task_failures
• Ssis_errors
• Transact_sql_errors
• sysdtslog90
Framework procs
Flow-Specific Tables
Work
Staging
Transformation_Error
Etl_control
Override
Thresholds
Logical_Errors
Staging_Hist
Override_Hist
Threshold_Hist
Corresponding procs

10. SSIS
• Parent and Child Templates
• Package naming convention
• Variable namespaces
• Data access methodology - procs
• Handling Task failures
• Handling detail error messages
• Handling proc failures
• Logging execution
• Pre-compiling script
• Unique system::id
• Compartmentalizing into many small packages vs one big
• Common flow
• Control the flow, let Sql operate on the data

11. Web Components - Operations

12. Web Components – Threshold Maintenance

13. ETL Success Notification

14. ETL Failure Notification

15. Web Components - Reporting

16. Thank You !
Mike Litchfield
Assistant Vice President – IT
Architect & Sr. Developer
Income Research & Management
mlitchfield@incomeresearch.com