Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Disciplined Agile Data Warehousing (DW)/Business Intelligence (BI)

3,733 views

Published on

Is your DW/BI team able to easily respond to changing requirements? Are your stakeholders ecstatic with what is delivered by your DW/BI team? Is your DW/BI team able to deliver new changes quickly into production? If the answer to any of these questions is no, then you likely want to consider adopting a disciplined agile approach to DW/BI development.

A disciplined agile DW/BI team will invest some time with initial requirements modeling, architecture modeling, and planning. But this will take a few days or maybe a week or two, certainly not months. They following a usage driven approach they will develop vertical slices of functionality. These slices fully tested and could potentially be deployed into production at any point in time. They work in an evolutionary and highly collaborative manner, resulting in quicker time to deliver, higher quality, and greater stakeholder satisfaction. This isn’t your grandfather’s traditional DW/BI approach.

Published in: Data & Analytics

Disciplined Agile Data Warehousing (DW)/Business Intelligence (BI)

  1. 1. Agile Data Warehousing and Business Intelligence: A Disciplined Approach
  2. 2. The Story I’m About to Tell © Disciplined Agile Consortium
  3. 3. We have many challenges to overcome together © Disciplined Agile Consortium
  4. 4. © Disciplined Agile Consortium
  5. 5. © Disciplined Agile Consortium
  6. 6. Modern development is evolutionary in nature © Disciplined Agile Consortium
  7. 7. You need to be flexible © Disciplined Agile Consortium
  8. 8. Collaboration is Crucial to Your Success © Disciplined Agile Consortium
  9. 9. © Disciplined Agile Consortium
  10. 10. We face several challenges © Disciplined Agile Consortium
  11. 11. Data Quality Capability © Disciplined Agile Consortium 16% 25% 15% 32% 10% 16% 29% 25% 24% 3% Very challenged Challenged Neutral Capable Very Capable Prevent New Problems Fix Existing Problems Source: 2015 Data Quality Survey Yikes!
  12. 12. Cultural Impedance Mismatch © Disciplined Agile Consortium • Developers • Have embraced evolutionary, and now agile and lean, strategies for over two decades • Are generally weak on data skills • Data professionals • Just now starting to embrace agile strategies for DW/BI • Are generally weak on development skills Source: Agiledata.org/essays/culturalImpedanceMismatch.html
  13. 13. Traditional DW/BI © Disciplined Agile Consortium • Heavy modeling performed early in the lifecycle • Heavy specification throughout lifecycle • Requirements changes are avoided • Solution released as a “big bang”
  14. 14. Disciplined Agile DW/BI © Disciplined Agile Consortium • High-level modeling performed early in the lifecycle • Light-weight modeling throughout the lifecycle • Comprehensive regression test suite(s) developed • Solution is developed in thin, vertical slices • Requirements changes are welcomed
  15. 15. © Disciplined Agile Consortium
  16. 16. © Disciplined Agile Consortium The Disciplined Agile Delivery (DAD) Framework
  17. 17. Disciplined Agile Delivery (DAD) is a process decision framework The key characteristics of DAD: – People-first – Goal-driven – Hybrid agile – Learning-oriented – Full delivery lifecycle – Solution focused – Risk-value lifecycle – Enterprise aware © Disciplined Agile Consortium
  18. 18. © Disciplined Agile Consortium
  19. 19. Scrum Extreme Programming LeanKanban DAD is a Hybrid Framework © Disciplined Agile Consortium Unified Process Agile Modeling Agile Data“Traditional”Outside In Dev. DevOps …and more DAD leverages proven strategies from several sources, providing a decision framework to guide your adoption and tailoring of them in a context-driven manner. SAFe
  20. 20. © Disciplined Agile Consortium
  21. 21. Which Lifecycle for DW/BI? • When the concept of DW/BI is completely new to your organization – The Exploratory lifecycle should be considered so that you can identify a potential strategy for the DW/BI solution • Initial release of a DW/BI solution – The Agile/Basic lifecycle is the typical choice – Straightforward approach to agile – Iterations force discipline onto a team that is likely new to agile • Future releases of a DW/BI solution – The Lean or Continuous Delivery lifecycles are the typical choices – New requirements are often small and independent of one another © Disciplined Agile Consortium
  22. 22. Basic/Agile Lifecycle A full Scrum-based agile delivery lifecycle. © Disciplined Agile Consortium
  23. 23. Lean Lifecycle A full lean delivery lifecycle © Disciplined Agile Consortium
  24. 24. Lean Continuous Delivery Lifecycle Your evolutionary end goal? © Disciplined Agile Consortium
  25. 25. Agile DW/BI Principles © Disciplined Agile Consortium
  26. 26. Agile DW/BI is Collaborative © Disciplined Agile Consortium NOT
  27. 27. Agile DW/BI is Usage Driven • Data-driven approaches are too narrowly focused • How people use, or work with, the data to fulfill business goals is the true issue focus on usage, not the underlying data structures • Work closely with your stakeholders to explore their intent © Disciplined Agile Consortium User stories drive development NOT data models
  28. 28. Agile DW/BI is Evolutionary © Disciplined Agile Consortium NOT
  29. 29. Agile Teams Work in Short Iterations © Disciplined Agile Consortium Source: SA+A 2015 Agile State of the Art Survey Disciplined agile DW/BI teams deliver new data or reports each iteration 1% 7% 15% 65% 13% > 4 weeks 4 weeks 3 weeks 2 weeks 1 week or less
  30. 30. Agile DW/BI Teams Strive to Fix Data at the Source • Extract Transform Load (ETL) is often a quality band-aid • Transformation is time consuming and risks defect injection – Often needed due to semantics differences • Disciplined agilists strive to fix data problems at the source via database refactoring – This is one aspect of paying down technical debt – Transformation is often dramatically reduced © Disciplined Agile Consortium
  31. 31. Agile DW/BI is Pragmatic • Sometimes: – You need to accept data inconsistencies – There is no “one truth” – You can’t get all of the data (at first) that you need for a report – Stakeholders can’t agree • Perfection is the enemy of good enough © Disciplined Agile Consortium Robert Smith Robert W. Smith Robert William Smith Bob Smith
  32. 32. Modern development is evolutionary in nature © Disciplined Agile Consortium
  33. 33. The Agile Database Techniques Stack © Disciplined Agile Consortium Vertical Slicing Clean Architecture and Design Agile Data Modeling Database Refactoring Database Regression Testing Continuous Database Integration Configuration Management
  34. 34. Vertical Slices of a DW/BI Solution • Every iteration the disciplined agile DW/BI team produces a working solution • Functionality is added in “vertical slices” • Slicing strategies: – One new data element from a single data source – One new data element from several sources – A change to an existing report – A new report – A new reporting view – A new data mart table © Disciplined Agile Consortium
  35. 35. High-Level Logical DW/BI Architecture © Disciplined Agile Consortium
  36. 36. Clean Database Design © Disciplined Agile Consortium • The cleaner the design of your database, the easier it is to evolve • Data Vault 2.0 is the best source of strategies for clean DW architecture and design
  37. 37. © Disciplined Agile Consortium
  38. 38. Agile Data Modeling • Data modeling is the act of exploring data-oriented structures • Evolutionary data modeling is data modeling performed in an iterative and incremental manner • Agile data modeling is evolutionary data modeling done in a collaborative manner © Disciplined Agile Consortium
  39. 39. © Disciplined Agile Consortium
  40. 40. Why Evolving Databases is Thought to be Hard Production databases are often highly coupled to other systems, services, data sources, … © Disciplined Agile Consortium
  41. 41. Database Refactoring • A database refactoring is a simple change to a database schema that improves its design while retaining both its behavioral and informational semantics • A database schema includes structural aspects such as table and view definitions; functional aspects such as stored procedures and triggers; and informational aspects such as the data itself © Disciplined Agile Consortium
  42. 42. The Process of Database Refactoring © Disciplined Agile Consortium
  43. 43. © Disciplined Agile Consortium
  44. 44. Database Regression Testing • Agilists develop regression unit and acceptance test suites for their applications, why not for databases too? • If database testing is done at all these days, it is at the black-box level. This isn’t enough • You must be able to put the database into a known state, therefore you need test data (generation) © Disciplined Agile Consortium
  45. 45. Test/Behavior Driven Database Development (TDDD/BDDD) • An evolutionary approach to database design • Driven by tests, not models • A practical approach to ensuring database quality • Completely different than traditional approaches to database design • Really just one part of BDD, not a stand-alone activity • May/June 2007 issue of IEEE Software © Disciplined Agile Consortium
  46. 46. © Disciplined Agile Consortium
  47. 47. Continuous Integration and Databases • Part of building the system is building the database (if it changed) • Challenge: Tests SHOULD put the database back into a known state, but sometimes don’t • You will want to rebuild the (non- production) database from scratch every so often • Challenge: Database accesses take time • Some test suites will test against DB mocks • You still need to test the actual database occasionally © Disciplined Agile Consortium
  48. 48. Configuration Management • All work products should be stored in a versioned repository • Maintains the integrity of the system and all supporting work products as it evolves • Facilitates change in a controlled fashion • All revisions kept and who made what changes to all work products • Potential benefits include the ability to: – Manage versions of systems across environments – Rollback to previous versions – Modify work products in parallel and then merge – Find source of defects injected © Disciplined Agile Consortium
  49. 49. The Story I Told © Disciplined Agile Consortium
  50. 50. We have many challenges to overcome together © Disciplined Agile Consortium
  51. 51. © Disciplined Agile Consortium
  52. 52. © Disciplined Agile Consortium
  53. 53. Modern development is evolutionary in nature © Disciplined Agile Consortium
  54. 54. You need to be flexible © Disciplined Agile Consortium
  55. 55. Collaboration is Crucial to Your Success © Disciplined Agile Consortium
  56. 56. © Disciplined Agile Consortium
  57. 57. Interested in Learning More? • We offer a hands-on workshop entitled Disciplined Agile Data Warehousing/Business Intelligence. Visit DisciplinedAgileConsortium.org/da210 for details. • At DisciplinedAgileConsortium.org/posters there is a downloadable poster overviewing the Disciplined Agile DW/BI approach • At DisciplinedAgileConsortium.org/webinars you can download a recorded presentation of this deck. © Disciplined Agile Consortium
  58. 58. Thank You! scott [at] scottambler.com @scottwambler AgileModeling.com AgileData.org Ambysoft.com DisciplinedAgileConsortium.org DisciplinedAgileDelivery.com ScottAmbler.com Disciplined Agile Delivery Disciplined Agile Delivery © Disciplined Agile Consortium

×