Introduction to the benefits of automating the testing of BI Solutions. Some samples with the open-source framework NBi. Automate the automation with genbi and genbiL.

1. Automating the testing of
your BI solutions with NBi
Cédric L. Charlier

2. Agenda
• Why automating is a must?
• Tests with queries (Sql/Mdx/Dax)
• Tests with ETL packages (SSIS)
• Testing multidimensional and tabular models (SSAS)
• Automating the creation of test-suites

3. What’s automated
testing?
A piece of software
written to reproduce
steps/actions defined in
a manual process

4. Benefits of automation
Allow testing to
happen more
frequently
Speed up
testing to
accelerate
releases
Improve test
coverage
Ensure
consistency

5. Tests automation
doesn’t apply to
Business Intelligence

6. Wednesday, November 24, 2012Est. 1869 Price 6d
More than 33% of BI
projects fail to deliver
Lack of
agility.
Aenean commodo
ligula eget dolor.
Aenean massa. Cum
sociis natoque
penatibus et magnis
dis Aenean commodo
ligula eget dolor.
Aenean massa. Cum
sociis natoqueCum
sociis natoqueCum
sociis natoque
Low quality of
deliverables
Aenean commodo ligula eget
dolor. Aenean massa. Cum
sociis natoque penatibus et
magnis dis parturientAenean
commodo ligula eget dolor.
Aenean massa. Cum sociis
natoqueCum sociis natoque
penatibus et magnis dis
parturient montes, nascetur
ridiculus mus. Donec quam
felis, ultricies nec,
Simply Too
late.
Cum sociis natoque
penatibus et magnis dis
parturient montes,
nascetur ridiculus mus.
Donec quam felis,
ultricies nec, Cum sociis
natoque penatibus et
magnis dis parturient
montes, nascetur
ridiculus mus. Donec
quam felis, ultricies nec,
Member of the Asscoiated Press .
Aenean commodo ligula eget dolor.
Aenean. Aenean commodo ligula eget
dolor. Aenhswse. Cejhciebce fcdcdcd.
ILLUSTRATED WEEKLY NEWSPAPER

7. We haven’t
time to create
these tests

8. Time to execute a test-suite
1.0 1.1 1.2 1.3 1.4 1.5
Elapsedtime
Release
Manual
Automate
Exponential
slope
Why?
- New objects
- Old objects

9. Time to write a test-suite
1.0 1.1 1.2 1.3 1.4 1.5
Elapsedtime
Release
Manual
Automated
With Framework
Initial gap

10. No existing framework
for test automation
dedicated to BI
… I’m not proficient
enough in C#

11. This framework
exists: NBi
No C#, just xml
syntax
Open-
source

12. Agenda
• Why automating is a must?
• Tests with queries (Sql/Mdx/Dax)
• Tests with ETL packages (SSIS)
• Testing multidimensional and tabular models (SSAS)
• Automate the creation of test-suites

13. Compare queries … result-sets
System-under-test Assert
Previous
release
Two different releases of
a same data warehouse
Tabular model versus Csv fileDAX
… or any brilliant idea
Cube and data
warehouseMDX SQL

14. Demo: compare queries’ result
• Compare cube/operational database
• 101
• Compare cube/flat files
• Defaults and References
• Tolerance
• Intervals

15. Other testing goals for queries
• Performance of a query
• Executed in less than x seconds
• Cache management
• Syntax of a query
• Extract queries from reports and assemblies

16. Query’s result and format
• Result’s format
• Sample: all the cells have effectively 2 digits after the decimal separator
and this separator is a comma.

17. Check calculations in your query
• Assert a logical test, row by row, for you result-set
• Sample: Calculate the total price from the unit price, quantity and
discount percentage.
• If at least one row doesn’t validate this logical test, the whole test shift
to red

18. Agenda
• Why testing automation is a must?
• Tests with queries (Sql/Mdx/Dax)
• Tests with Etl packages (SSIS)
• Testing multidimensional and tabular models (SSAS)
• Automate the creation of test-suites

19. Demo SSIS
• Assert the execution status of an ETL
• Alternatively test performances

20. Inside this ETL
Dimension
City
Dimension
Scientist
Factless fact
Birth/Death
insert insertUnique
index
violation

21. Setup and Cleanup
• Setup:
• Define the state of your testing environment before executing a test
• Cleanup:
• Remove consequences of your tests on the testing environment

22. Tasks and behaviors
• Available tasks
• Truncate/Load tables with csv files
• Run sql batch, exe or batches
• Run Etl packages
• Move or delete files
• Start/Stop Windows services
• Run-once: execute only once this set of tasks
• Parallel: run a set of tasks in parallel

23. Demo Setup
• Setup to register a clean state before test’s execution
• Assert database’s state after the execution of an ETL

24. Agenda
• Why automation is a must?
• Testing queries (Sql/Mdx/Dax)
• Testing Etl packages (SSIS)
• Testing multidimensional and tabular models (SSAS)
• Automate the creation of test-suites

25. Testing dimension’s members
• “Germany” is a member of dimension “Country”
• “Europa” is not a member of dimension “Country”
• Dimension “Country” has more than 190 members.
• All members of dimension “Country” have at least 4 characters
• Members of hierarchy “Month” are ordered chronologically (and
not alphabetically)
• …

26. Testing structure
• Assert that a dimension, hierarchy, measure, … is visible for end-
user in a given perspective.
• Assert the existence of a relation between a measure-group and
a dimension.

27. Agenda
• Why automation is a must?
• Tests with queries (Sql/Mdx/Dax)
• Tests with Etl packages (SSIS)
• Testing Multidimensional and tabular models (SSAS)
• Automating the creation of test-suites

28. Some tests are
really repetitive
It should be
possible to
automate the
creation

29. It exists! Take a
look at Genbi
A template for
the tests
A set of test-
cases

30. First steps with Genbi
File loaded
from Genbi
Variables to be
substituted
(surrounded by $)

31. Demo: Genbi
• Load test-cases
• From a flat file
• Select a template
• Generate the test-suite
• Grouping option

32. I’m scripting macros to
automate the actions
performed by genbi …
With genbiL!

33. Sample for genbiL
Subject Action Parameters

34. Demo for genbiL
• Command line

35. Conclusions
• Benefits of test automation
• Save Time (Time-on-market)
• Better Quality (end-result)
• More confidence
• NBi
• Support large range of different tests
• No need of compiler (neither C#)
• Good feedback when tests are failing
• Open-Source
• Automate the creation of your test-suites (genbi / genbiL)

36. Read further
• Website: http://nbi.codeplex.com
• Download, documentation,
bugs/Wishes list, support
• Blog: http://seddryck.wordpress.com
• Twitter: @Seddryck
Partnership
TestingSquad
Rue des Déportés, 47
1200 Bruxelles
phone : +32 470/84.07.47
email : daniel.mueller@testingsquad.eu
Web : www.testingsquad.eu