This document discusses using Apache JMeter to test ETL processes. It provides examples of common ETL tests like counting rows and keys, checking for duplicate loads, and validating metrics over time. The document notes challenges with current manual SQL querying and custom code-based approaches. It proposes using Apache JMeter as a solution since it is free, platform independent, connects to different databases, and provides an easy GUI and XML reports. A demo is included to show the test structure, results transformation to XML, and report transformation capabilities of Apache JMeter.

1. ETL testing accelerator
Using Apache jMeter
Novostavskyy V.
Epam systems

2. Short reminder what’s ETL
extractsources transformstaging loadDWH

3. What can go wrong?
In general it can be that data …
 should be loaded and is not loaded
 should not be loaded and is loaded
 is loaded with duplications (multiplied)
 Is loaded but corrupted (wrong transformation is applied)

4. Test examples
 Count keys, count rows
 Check BK 1:1 load
 Gather metadata metrics over time
 Count distinct values for some dims (e.g. distinct salesmen)
 Sum of measurable metric (e.g. earned money)
 Write code that makes the same transformation
 Apply “smart tests” based on the nature of your data (from the business)

5. Problem
 Every test – sql query, but…
 Query are run manually, that’s frustrating, you have issues with reports to
clients/team
 Or you have to write .net/java/python code (which is nice, of course)
 Or you have nice tool implemented on ETL platform, say MS SSIS, but your
next project uses Oracle/PostgreSQL/smth else and oyu need to reinvent it
for new tech stack

6. Nice to get
 Multi-platform test runner
 With result assertion
 And results reporting
 That I can start using from the first day
 And I shouldn’t know python/java/js to manage it

7. Proposed solution
Apache Meter
 Free software (Apache 2.0)
 Run anywhere where you have jvm
 Connects to different DBs over JBC
 Easy to manage tests and suits in GUI
 Able to run agents directly on server side, or from your local machine
 Nice xml reports, transformed into html

8. Demo

9. Test structure
 Variable and parametrization
 Thread sequence
 Thread names
 If conditions
 Sampler names
 Assertions

10. Results transformation
 Created XML file
 Easy to transform into HTML (implemented)

11. Report transformation

12. Questions