ETL Metadata Injection with Pentaho Data Integration (Meetup Pentaho Madrid 26th January 2015)

1. ETL Metadata Injection
David Fombella Pombal
http://anonymousbi.wordpress.com
@Pentaho_Fan

2. ETL Metadata Injection 2
David Fombella Pombal
BI Consultant, Technical Developmental Editor and
Pentaho Community Guy
About me

3. ETL Metadata Injection 3
Introduction
•The ETL Metadata Injection step is capable of injecting
step metadata into a template transformation. So
instead of statically entering ETL metadata in a step
dialog, you pass it at run-time. It is possible to solve
repetitive ETL workloads like loading of text files, data
migration and so on.

4. ETL Metadata Injection 4
Problem description
•We need to load cost files containing a dynamic header
•Budget files contain 12 months information
•Forecast files contain between 1 and 12 months
•3 fixed columns containing Entity, Account and Cost Centre

5. ETL Metadata Injection 5
File Types
•Budget file containing 2015 (12 months)
•Forecast file October 2014 (3 months)

6. ETL Metadata Injection 6
Easy to use +
Open Source +
BI Solution =
Pentaho and
Mondrian!
Template transformation (Text file input)

7. ETL Metadata Injection 7
Easy to use +
Open Source +
BI Solution =
Pentaho and
Mondrian!
Template transformation (Row normaliser)

8. ETL Metadata Injection 8
Load file Metadata Injection 1

9. ETL Metadata Injection 9
Load file Metadata Injection 2

10. ETL Metadata Injection 10
Load file Metadata Injection 3

11. ETL Metadata Injection 11
Load file after Injection

12. ETL Metadata Injection 12
Non-Native Metadata Injection
Metadata Injection is one of the single most powerful
features of PDI. However it only some steps support it. But
as PDI has 100s of steps, more being added weekly, there’s
always going to be a scenario where the step you want to
Inject doesn't support it.
So what do you do? Use the API. The API is how PDI itself
sets the metadata for a step.
So the steps are:
1. Open the transformation file
2. Find the step(s) we want to change
3. Configure it accordingly
4. Save the file

13. ETL Metadata Injection 13
Non-Native Metadata Injection (template)

14. ETL Metadata Injection 14
Non-Native Metadata Injection (Inject 1)

15. ETL Metadata Injection 15
Non-Native Metadata Injection (Inject 2)

16. ETL Metadata Injection 16
Non-Native Metadata Injection (Inject 3)

17. ETL Metadata Injection 17
Non-Native Metadata Injection
Thanks to…