The document outlines best practices for working with Pentaho Data Integration (PDI), including naming conventions, parameters, logging, performance optimization, and design patterns. It recommends meaningful naming, keeping the environment tidy, using variables, logging everything, optimizing slow steps, automating lifecycles, and demonstrates loops, queues, load balancing, watchdog, and auto-recovery patterns. The presentation aims to improve quality, simplify processes, and save time for data integration work.

1. Pentaho Data Integration
Best Practices
Matt Casters
Chief Data Integration, Kettle Founder
#PCM14

2. © 2014, Pentaho. All Rights Reserved.
2 #PCM14
About this session
Today You Will Learn
Best practices when working
with Pentaho Data
Integration
Agenda
❯Working with PDI
❯Design patterns
❯Demo fun stuff

3. © 2014, Pentaho. All Rights Reserved.
3 #PCM14
Working with PDI

4. © 2014, Pentaho. All Rights Reserved.
4 #PCM14
Working with PDI: Naming
❯Provide meaningful names for steps and job entries
❯Do not hesitate to use special characters

5. © 2014, Pentaho. All Rights Reserved.
5 #PCM14
Working with PDI: Naming
❯Avoid environment specific names
Test Database
France MySQL
East Coast Cluster
CRM
WWW
Cluster

6. © 2014, Pentaho. All Rights Reserved.
6 #PCM14
Working with PDI: Naming
❯Keep your environment tidy
• Folders can have sub-folders!
❯Use naming conventions for everything
• Database tables and fields
• Directories
• Server names

7. © 2014, Pentaho. All Rights Reserved.
7 #PCM14
Working with PDI: Naming
❯Use a corporate standard
❯Verify and enforce periodically
❯Use rules to validate repository imports
• Database names
• Notes and descriptions

8. © 2014, Pentaho. All Rights Reserved.
8 #PCM14
Working with PDI: Naming
❯Use a corporate standards
❯Verify and enforce periodically
❯Use rules to validate repository imports
• Database names
• Notes and descriptions

9. © 2014, Pentaho. All Rights Reserved.
9 #PCM14
Working with PDI: Tidy up!
❯Limit the number of steps or job entries

10. © 2014, Pentaho. All Rights Reserved.
10 #PCM14
Working with PDI: Tidy up!
❯Enable grid size 32 or 16
❯Prevents accidental move of step or entry

11. © 2014, Pentaho. All Rights Reserved.
11 #PCM14
Working with PDI: Parameters
❯Explicit use of variables
❯Easier testing
❯Make re-use a breeze

12. © 2014, Pentaho. All Rights Reserved.
12 #PCM14
Working with PDI: Variables
❯Environment specific : kettle.properties
❯Prefer ${SOLUTION_HOME}
❯Avoid ${Internal.Transformation.Filename.Directory}
❯Configure step copies with variables

13. © 2014, Pentaho. All Rights Reserved.
13 #PCM14
Working with PDI: Logging
❯Log everything!
❯Measurement is management
❯Use the Pentaho audit mart
❯Learn about all the possible logging features

14. © 2014, Pentaho. All Rights Reserved.
14 #PCM14
Working with PDI: Mappings
❯Mapping vs Simple Mapping step
❯Realize this is a macro
❯Use completely different field names
❯Avoid renaming or removing fields

15. © 2014, Pentaho. All Rights Reserved.
15 #PCM14
Working with PDI: Metadata Injection
❯Avoid manual population of dialogs
❯Whenever you need dynamic ETL
❯5.1 supports data streaming
❯Example:
• stage 50 different files with one transformation

16. © 2014, Pentaho. All Rights Reserved.
16 #PCM14
Working with PDI: Performance
❯Transformations are networks
❯Network speed is limited to the slowest part
❯The slowest step is indicated while running in Spoon
❯Slow steps have a full input and empty output buffer

17. © 2014, Pentaho. All Rights Reserved.
17 #PCM14
Working with PDI: Performance
❯First re-write, re-think, re-organize
❯Parallize work
❯End-to-end data pipe-lining
❯Do work where it's the fastest (ELT)

18. © 2014, Pentaho. All Rights Reserved.
18 #PCM14
Working with PDI: Lifecycle
❯Automate export of repositories
❯Use import rules to validate quality
❯Always use version control for file based setup

19. © 2014, Pentaho. All Rights Reserved.
19 #PCM14
Design patterns

20. © 2014, Pentaho. All Rights Reserved.
20 #PCM14
Design patterns: Loops
❯Use the job or transformation executor steps
❯Much easier since version 5
❯Demonstration: process lots of small files

21. © 2014, Pentaho. All Rights Reserved.
21 #PCM14
DEMONSTRATION

22. © 2014, Pentaho. All Rights Reserved.
22 #PCM14
Design patterns: Queues
❯Process buffer
❯Facilitates parallelism
❯Forces process logging best practice
❯Only way to process recurring files

23. © 2014, Pentaho. All Rights Reserved.
23 #PCM14
Design patterns: Load balancing
❯On a step level: PDI EE 5.x built-in
❯Balance jobs and transformations with carte
❯Set up a carte cluster
❯Use a queue
❯Interrogate Carte web services

24. © 2014, Pentaho. All Rights Reserved.
24 #PCM14
DEMONSTRATION

25. © 2014, Pentaho. All Rights Reserved.
25 #PCM14
Design patterns: Watchdog
❯“Who watches the watchmen?”
❯Simple recipe:
• On success increment a counter
• Periodically verify that the counter is advancing
• Take action when counter is not advancing

26. © 2014, Pentaho. All Rights Reserved.
26 #PCM14
Design patterns: Watchdog
❯Schema
Main Job
Job Counter
Success?
Increase counter
Watchdog
Validate that
counter
increased
Action
Action when
not increased

27. © 2014, Pentaho. All Rights Reserved.
27 #PCM14
Design patterns: Auto-recovery
❯Auto-skip:
• use anything incremental
• Add incremental ID to source tables if missing
❯Auto-cleanup:
• increment run ID after successful job
• Remove run ID from target table at start of job
❯Database recovery:
• Job and transformation level transactions (PDI 5 EE)

28. © 2014, Pentaho. All Rights Reserved.
28 #PCM14
Fun stuff...
a short demo to wrap up

29. © 2014, Pentaho. All Rights Reserved.
29 #PCM14
Summary
To take away:
❯Best practices improve quality, simplify, save time
❯Use this presentation as a checklist
❯Do regular audits of your data integration work