BI went Agile, but Testing didn’t. Presented by Stefaan De Vos (Automade)

1. Automade
Test Automation
Data Vault and Data
Warehouse Automation
9th of December
Stefaan De Vos

2. Test Automation

3. Challenges
• Lead times of BI / Analytical projects are ever
decreasing
• Tools
• Automation (Wherescape...)
• Virtualization (Denodo, CIS...)
• Appliances (Netezza, DATAllegro...)
• Methodologies
• Data Vault
• Agile DWH
BI went Agile, but Testing didn’t

4. Challenges
• Data
• Larger data volumes
• IoT
• Unstructured data / poor data quality
• Social networks
• No relevant test data sets available
• The number of possible test cases are near
infinite
Data Volume & Quality

5. Challenges
Review
requirements
Review
requirements
Define testing
Strategy
Define testing
Strategy
Prepare Test DataPrepare Test DataEntry / Exit criteria
Design Test Cases &
Scripts
Design Test Cases &
Scripts
Configure the test
environment
Configure the test
environment
Prepare & Design
Agree on Entry / Exit
test criteria
Agree on Entry / Exit
test criteria
Integration
testing
Integration
testing
Performance testingPerformance testing
Acceptance testingAcceptance testing
- Basic testing
- DI jobs accessible
- Reports accessible
- Cleansed
- Complete
- Correct
- Integrated
- Valid reports
- Relevant data
- Available data
- Consistent data
- Accessible
- DI & BI integration
- Full test cycles
- Check NFRs
- Scalable
- Check SLAs
- Peak user
- Peak loads
- Functional
- User
- Production
- SME
- End user
Construct
Defect metrics
review
Defect metrics
review
Performance
statistics
Performance
statistics
Lessons learntLessons learnt
Process & Quality Improvement
Accept
Data Completeness
testing
Data Completeness
testing
BI & Analytical
Testing
BI & Analytical
Testing
Smoke test
Unit test
Smoke test
Unit test
BI Testing Lifecycle

6. Challenges
** Regulatory compliance might require to use the v-model, e.g. validated environments.
Functional
analysis
Requirements
functional
Non-functional
Technical
Design
Construction
Unit
Smoke Test
Data
Completeness
BI &
Analytical
Integration
testing
Performance
User
Production
AcceptanceValidation
Verification
BI Testing V-Model

7. Challenge
• Lots of data
• Lots of testing to be done
• Little time to do it
Summary

8. Test Automation

9. Complete DWH Testing
• Complete testing is a must
• It requires:
• Testing methodology
• Right project culture/mindset and organization
• Tools
Our View

10. Complete DWH Testing
Database
integrity
checking.
Risk based
testing
Effective defect
management
and
collaboration
End to End
Performance
testing
Adherence to
compliance and
regulatory
standards
and…AUTOMATE
Critical Success Factors

11. Test Automation
Leverage social development principles to deepen
functionalities
Testers
- functional testing
- regression testing
- result analysis
Developers / DBAs
- unit testing
- result analysis
Data Analysts
- review, analyze
data
- verify mapping
failures
Operations teams
- monitoring
- result analysis
Collaboration/WorkflowTestManagement
Rational Quality Manager JIRA Team Foundation ServerQuality Center

12. Test Automation

13. What to Automate?
Complex
Functional
SQL
Validation
Reconciliation

14. Test Automation
Basic functionality
• Auto detection of anomalies (or at least prior to being detected by a user)
• Targeted regression testing for planned changes
• Data error identification errors by comparing (huge) data result sets
• Data error detection via a rules engine
• In between data layer reconciliation and auditing
• Test data generation
Additional functionality
• No programming or coding
• Heterogeneous connectivity
• Collaboration and workflow capabilities
• Visually attractive development and monitoring environment
• Intuitive reports & dashboards
Functionality

15. Test Automation
• Shorten regression cycles
• Save report developers time
• Test the same data set in less time
• Test more
• Faster deployment of defect resolution cycle
• Faster deployment of enhancements
• Less cumbersome upgrades/migrations
• Enabler for
• Implementing continuous testing
• operationalization of testing
Benefits

16. Test Automation
Vendors - tools
• ICEDQ
• RTTS -
QuerySurge
Unconnected ETL
Source Data Layer Target Data Layer
ETL - ELT
1 3
2
Load test data
Execute Job externally
Extract result
4 Compare & report
• Validation of the business rules
implemented in ETL processes
are assessed by comparing the
results against a ground truth.
• The ETL processes are executed
separated from the test
automation tool
• Less adequate for multi-step ETL
processes.
Approach

17. Test Automation
Vendors - tools
• Zuzena
Connected ETL
• The business rules present
in the ETL tool are analyzed
and the test results are
assessed against the
anticipated results.
• The test automation tool
executes the ETL processes.
Approach
Source Data Layer Target Data Layer
2 4Load test data Extract result
5 Compare & report
1
Analyze logic
3
Run job

18. Test Automation
Vendors - tools
• Report Valid8tor (BO)
• 360Bind (BO)
• Integrity Manager
(MSTR)
• Report Validator - BSP
Software (Cognos)
Report Integrity validator
• Parallel testing of 2 live
systems
• Comparing against a
(historical) ground truth
• Comparing against
known good baselines
Approach
Reports
Reporting data Layer
Load test data Scrape data
4 Compare & report
1 3
Run report
2

19. Test automation

20. QuerySurge
• Integrates with HP QC / IBM RQM / MSFT
TFS
• Provides collaboration features
pulls data from data sources
pulls data from target data store
compares data quickly
generates reports, audit trails
reports
SQL
Design Tests
Scheduling
Reporting
Run
Dashboard
Wizards
Data
Health
Dashboard
• a SQL execution and data comparison tool
working against heterogeneous
datasources.
• No programming needed

21. Automade introduction
• Automade
• Provides an agile answer to the ever-increasing
information appetite
• By automating the mind-numbing aspects of
constructing, maintaining and testing of data
warehouses.
• Automade is a spinoff of MindThegap and
has established partnerships with

22. Test automation
Thank you for listening,
Any questions?
Feel free to send questions & feedback to stefaan.devos@automade.be