Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to Lynn Winterboer : Test automation
Similar to Lynn Winterboer : Test automation (20)
More from AgileDenver
More from AgileDenver (20)
Recently uploaded
Recently uploaded (20)
Lynn Winterboer : Test automation
- 1. © 2016 by Winterboer Agile Analytics www.WinterboerAgileAnalytics.com Test Automation: Agile Enablement for Data Warehousing and Business Intelligence Teams Presented to Mile High Agile (MHA) 2016 April 4, 2016 Denver, Colorado Lynn Winterboer, Agile Analytics Educator and Coach, @AgileLynn Brad Ewald, DW/BI Solution Architect, @JBEwald
- 2. © 2016 by Winterboer Agile Analytics www.WinterboerAgileAnalytics.com Agenda • Why is test automation important for agile data teams? • Why aren’t all data teams automating their tests today? • What is the path to test automation? • What does simple DW test automation look like?
- 3. © 2016 by Winterboer Agile Analytics www.WinterboerAgileAnalytics.com Why is test automation important to agile data teams?
- 4. © 2016 by Winterboer Agile Analytics www.WinterboerAgileAnalytics.com© Ken Collier, 2016, All Rights Reserved Agile Demands Something Different Agile calls for small increments of “potentially shippable code”. That means QA is essential on a regular, frequent basis. How can we be sure that something we do here Doesn’t break something we did back here Test Suite Test Suite Test Suite Test Suite Test Suite Test Suite Test Suite Manual testing quickly becomes infeasible. The test suite grows larger each iteration.
- 5. © 2016 by Winterboer Agile Analytics www.WinterboerAgileAnalytics.com© Ken Collier, 2016, All Rights Reserved Testing is Central to Agile Agile BI development is driven by tests Acceptance criteria are the definition of “done” Passing tests are the measure of “done” Regression tests are the measure of “still done”
- 6. © 2016 by Winterboer Agile Analytics www.WinterboerAgileAnalytics.com© Ken Collier, 2016, All Rights Reserved Agile Teams Understand… … that test automation is a key technical enabler to “being agile”
- 7. © 2016 by Winterboer Agile Analytics www.WinterboerAgileAnalytics.com Why aren’t all data teams automating their tests today?
- 8. © 2016 by Winterboer Agile Analytics www.WinterboerAgileAnalytics.com DW/BI Test Automation Challenges Data industry does not focus on testing as much as other disciplines: Education Team Members Skills & Discipline
- 9. © 2016 by Winterboer Agile Analytics www.WinterboerAgileAnalytics.com DW/BI Test Automation Challenges Existing tools are not easily adapted to the data world: Development Languages Web, Mobile and UI Focus
- 10. © 2016 by Winterboer Agile Analytics www.WinterboerAgileAnalytics.com DW/BI Test Automation Challenges Good test data sets are hard to come by: Large volumes (on small environments) Shared test environments Data sensitivity Small sets need extra planning and design
- 11. © 2016 by Winterboer Agile Analytics www.WinterboerAgileAnalytics.com What is the path to test automation?
- 12. © 2016 by Winterboer Agile Analytics www.WinterboerAgileAnalytics.com© Ken Collier, 2016, All Rights Reserved Agile Testing Perspectives High Quality DW/BI System 3) Tools & Automation 1) Attitudes & Culture 2) Skills & Practices Testing is a Team Sport Learning Culture Build Quality In Testing Skills Test Design, Organization & Maintenance Repeatable Tests Feedback Loops Test Environment Build Automation Test Data Sets Test Automation Tool
- 13. © 2016 by Winterboer Agile Analytics www.WinterboerAgileAnalytics.com© Ken Collier, 2016, All Rights Reserved Test Automation Pyramid App Tests Acceptance Tests Unit & Component Tests Manual Tests Lisa Crispin & Janet Gregory, Agile Testing: A Practical Guide for Testers and Agile Teams Push as many tests as possible to this layer Write these in business domain language Brittle tests that are affected by UI changes Subjective user feedback is essential
- 14. © 2016 by Winterboer Agile Analytics www.WinterboerAgileAnalytics.com© Ken Collier, 2016, All Rights Reserved Good Places to Start Unit tests on new development Regression tests you wish you had, before building something new Deployment tests to ensure each build migrated correctly Painful test setup or test execution prone to error
- 15. © 2016 by Winterboer Agile Analytics www.WinterboerAgileAnalytics.com What does simple data warehousing test automation look like?
- 16. © 2016 by Winterboer Agile Analytics www.WinterboerAgileAnalytics.com© Ken Collier, 2016, All Rights Reserved Basic Approach Component under test Source Schema Target Schema Actual Results Test Data Test Runner Setup Execute Test Report Clean Up Static Test Data Expected Results Test Queries load test data compare results execute tests
- 17. © 2016 by Winterboer Agile Analytics www.WinterboerAgileAnalytics.com Demonstration
- 18. © 2016 by Winterboer Agile Analytics www.WinterboerAgileAnalytics.com Testing Framework Functionality and Components: Example data set SQL Server and related model Form for creating and describing tests Microsoft Access form Test execution SQL Server stored procedures (individual tests) Power Shell scripts (batch execution) BI Reports and Dashboards for monitoring MS Power BI
- 19. © 2016 by Winterboer Agile Analytics www.WinterboerAgileAnalytics.com What Constitutes a Test? Actual Result (or Dataset) Expected Result (or Dataset) Always a Query SQL Query or Static Value
- 20. © 2016 by Winterboer Agile Analytics www.WinterboerAgileAnalytics.com© Ken Collier, 2016, All Rights Reserved Test Scenarios 1. Record Comparison 2. Sales Revenue Subject Area Test 3. Operational Data Quality Test
- 21. © 2016 by Winterboer Agile Analytics www.WinterboerAgileAnalytics.com Testing Demo
- 22. © 2016 by Winterboer Agile Analytics www.WinterboerAgileAnalytics.com Session Intent 1) Why: Test Automation is key to DW/BI agile success 2) How: Automate a solid testing practice; don’t automate chaos 3) Show: DW/BI test automation is not that hard
- 23. © 2016 by Winterboer Agile Analytics www.WinterboerAgileAnalytics.com DW/BI Test Automation Tools Tool Description DBFit http://dbfit.github.io/dbfit/ Open source database testing tool iCEDQ http://icedq.com A unified platform for ETL testing and data migration testing Analytix Data Services http://analytixds.com Enterprise data mapping and ETL code automation, including new automatic generation of reusable test cases associated with Source to Target Mappings WhereScape https://www.wherescape.com Data warehouse automation software, including a test harness framework with user-configurable profiling methods. Tricentis Tosca http://www.tricentis.com/tricentis- tosca-testsuite/ DW-friendly test automation tool
- 24. © 2016 by Winterboer Agile Analytics www.WinterboerAgileAnalytics.com DW/BI Test Automation References Reference Description Agile Analytics: A Value-Driven Approach to Business Intelligence and Data Warehousing by Dr. Ken Collier Chapter 7: Test-Driven Data Warehouse Development Agile Data Warehousing for the Enterprise: A Guide for Solution Architects and Project Leaders By Ralph Hughes Part V: Agile EDW Quality Management Planning Automated Data Warehouse Testing: Beginner’s Step-by- Step Guide by G. Suden Java-based test automation for simple data warehousing tests Agile Testing: A Practical Guide for Testers and Agile Teams by Lisa Crispin and Janet Gregory Great introduction to agile testing for any type of development More Agile Testing: Learning Journeys for the Whole Team by Lisa Crispin and Janet Gregory Includes several chapters on test automation as well as one dedicated to agile testing for DW/BI
- 25. © 2016 by Winterboer Agile Analytics www.WinterboerAgileAnalytics.com Thank You! Lynn Winterboer Agile Analytics Educator & Coach www.WinterboerAgileAnalytics.com lynn@WinterboerAgileAnalytics.com @agilelynn Brad Ewald DW/BI Solution Architect Children’s Hospital Colorado brad.ewald@gmail.com @jbewald
Editor's Notes
- Bring: Big stickies with markers for the parking lot Business Cards Laptop and power cord
- Agile requires that we include QA activities in every iteration to verify that we are building shippable quality code. Agile requires that we include regression testing in every iteration to ensure that we aren’t breaking previously working components. Agile requires that we automate our tests so that regression testing doesn’t become a development bottleneck.
- Effective Agile testing involves 3 key factors: Tools for automated testing – This is relatively easy to adopt and learn. Typically takes a technical team less than a week to stand up this environment. Principles and practices for effective testing – This requires discipline and commitment, and is more challenging than tools and automation. A culture and attitude that values high quality in everything – This is the hardest factor to “teach” and to change.
- Unlike conventional testing, which relies heavily on system and integration testing, Agile testing is principally supported by unit and component testing. Unit & Component Testing – The technology-facing tests that support the development team Made up of robust and fully automated unit and component tests Highest percentage of automated tests Written in the same language as the system under test (e.g., SQL) Customers do not need to be able to read these tests These tests are by far the quickest and easiest to write They provide the quickest feedback, making them highly valuable These tests have the highest ROI by far! Acceptance Tests - The business-facing tests that support the development team by testing underneath the GUI Functional tests that validate that we are building the right thing Tests that cover broader chunks of business meaning or semantics These tests are easier to write since they are underneath the BI application layer These tests generally run more slowly since they cover more ground than unit tests Try to write these tests in a domain specific language (not SQL) so that customers can understand them. This takes more time. The feedback is quick, but not as quick as unit tests. Therefore ROI is moderate. BI Application Tests - The business-facing tests that validate that we’ve built the right product by testing the user experience Testing through the UI to validate the proper behavior and data under all usage scenarios More expensive to write and adversely affected by changes to the UI These tests tend to be written after development is complete These tests are slower to run since they rely on the UI layer These test provide a lot of value, but a relative low ROI Manual Tests - Actual users performing exploratory testing on the BI applications Subjective but invaluable mechanism for customer feedback ROI is high as long as the other layers are robust, otherwise ROI is low Users gain confidence in data validity through lower layers, and provide usability feedback at this layer
- The fundamental steps in database test automation process are: Load a fixed set of test data Run the process that is under test Verify the result set Return everything to the way you found it Database test automation involves these elements: The component under test is generally an ETL module, SQL script, stored procedure, or other functional component. The static test data is carefully constructed, as small as possible, and stored in version control. The test queries and expected results are constructed with respect to the static test data. The test runner handles everything else.