Agile BI& Data Virtualization                            Tom Breur          tombreur@xlntconsulting.com                   ...
It’s a stretch… Volumes of data are growing (fast)! Variety of sources keeps expanding:   Social media, RFID, log-files...
Why go “Agile”? (1) BI projects fail too often, or don’t live up to  expectations Increasingly, BI development takes pla...
Why go “Agile”? (2) Winston Royce (1970):         “In my experience, the simpler model …      [as pictured below] has neve...
History of development methods         Jackson (1975) “Principles of Program Design”                                      ...
Quick & Dirty ≠ Agile (1) www.agilemanifesto.org (principle #1):    “Our highest priority is to satisfy the   customer th...
Quick & Dirty ≠ Agile (2)      Top-down project management                 (e.g.: Scrum)                       &     Botto...
Quick & Dirty ≠ Agile (3)
BI requirements Information products trigger change  requests:  new data ⇒ insights ⇒ new requirements Gerald M. (Jerry)...
BI: means and ends uncertaintyMeans uncertainty                    Ends uncertainty How do we get there?                ...
Waterfall ⇔ Agile Waterfall/          Traditional                                                  Agile          Requirem...
Weinberg on Quality        “If quality isn’t an objective (if the software doesn’t have to work),   you can satisfy any ot...
Concurrent development (1) Waterfall: you can avoid mistakes/rework  by getting good requirements upfront The most costl...
Concurrent development (2) Agile: decide at “last responsible moment”   decisions that haven’t been made, don’t ever    ...
Inmon ⇔ Kimball (1)    3-tiered          2-tiered
Inmon ⇔ Kimball (2)Problems with Inmon                  Problems with Kimball Uncovering the ‘correct’            Smalle...
3NF ⇔ Dimensional (1)   www.xlntconsulting.com   17
3NF ⇔ Dimensional (2)                                           this problem gets (much!)                                 ...
Hyper normalized model                                 business keys,                                 context attributes (...
3-tiered DWH architecture                                       Metadata  Legacy                                          ...
Horses for courses 3NF   quickly & accurately capture transaction data   easy to get data in Hyper normalized   integ...
Backroom ⇔ Frontroom Back room                             Metadata                                                  Front...
Divide & Conquer “Break down” semantic gap from back- to  front room Offer a range of data services:   Source data “as ...
Why data virtualization? Operational BI calls for real-time data Integrate heterogeneous sources, at least  “in the eye ...
Conclusion Big Data are here to stay  (and lets hope the hype passes soon) Data provide a source of sustainable  competi...
Conclusion Big Data are here to stay  (and lets hope the hype passes soon) Data provide a source of sustainable  competi...
Conclusion Big Data are here to stay  (and lets hope the hype passes soon) Data provide a source of sustainable  competi...
Upcoming SlideShare
Loading in …5
×

Tom Breur, XLNT - Agile BI And Data Virtualization - BI Symposium 2012

808
-1

Published on

0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
808
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
62
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide
  • Agile BI & Data Virtualization 26-NOV-2012 DWH Automation conference
  • Agile BI - How it works and ho to do it 6-NOV-2012 www.xlntconsulting.com
  • Agile BI - How it works and ho to do it 6-NOV-2012 DWH Automation conference
  • Agile BI - How it works and ho to do it 6-NOV-2012 DWH Automation conference
  • Agile BI & Data Virtualization 26-NOV-2012 www.xlntconsulting.com
  • Agile BI & Data Virtualization 26-NOV-2012 www.xlntconsulting.com
  • Agile BI & Data Virtualization 26-NOV-2012 www.xlntconsulting.com
  • Agile BI & Data Virtualization 26-NOV-2012 www.xlntconsulting.com
  • Agile BI & Data Virtualization 26-NOV-2012 www.xlntconsulting.com
  • Tom Breur, XLNT - Agile BI And Data Virtualization - BI Symposium 2012

    1. 1. Agile BI& Data Virtualization Tom Breur tombreur@xlntconsulting.com BI & IM Symposium Bussum, 26 November 2012
    2. 2. It’s a stretch… Volumes of data are growing (fast)! Variety of sources keeps expanding:  Social media, RFID, log-files, GPS, etc. Business users need their data (much) sooner:  monthly ⇒ weekly ⇒ daily ⇒ intra-day BI in support of operational processes, calls for (near) real-time data
    3. 3. Why go “Agile”? (1) BI projects fail too often, or don’t live up to expectations Increasingly, BI development takes place alongside (instead of a fte r) application engineering
    4. 4. Why go “Agile”? (2) Winston Royce (1970): “In my experience, the simpler model … [as pictured below] has never worked on large software development efforts” Analysis Design Development Test Release [Royce subsequently went on to describe an enhanced model, whichincluded building a prototype first and then using the prototype plus feedback between phases to build a final deployment] www.xlntconsulting.com 4
    5. 5. History of development methods Jackson (1975) “Principles of Program Design” Martin (1991) “Rapid Application Development” Anderson (2010) “Kanban” Boehm (1986) “A Spiral model of Software Development and Enhancement” Beck (2000) “Extreme Programming Explained” Brooks (1975) “The Mythical Man Month” Poppendieck (2003) “Lean Software Development”1960 1970 1980 1990 2000 2010 1994: DSDM Consortium launchedRoyce (1970) “Managing the Development of Large Software Systems” Cockburn (2004) “Crystal Clear” 1997: term eXtreme Programming (XP) ‘invented’ 1996: Scrum ‘invented’ 2001: term ‘Agile’ adopted From “code and fix” to more structured, methodical approaches to software development Unstructured Prescriptive methods Structured methods www.xlntconsulting.com 5
    6. 6. Quick & Dirty ≠ Agile (1) www.agilemanifesto.org (principle #1): “Our highest priority is to satisfy the customer through early and continuous delivery of valuable software” [emphasis added] Creating “technical debt” stands squarely in the way of continuous delivery, and maintaining a so-called “sustainable pace”:
    7. 7. Quick & Dirty ≠ Agile (2) Top-down project management (e.g.: Scrum) & Bottom-up software engineering (e.g.: Extreme Programming - XP) ⇓ Expedited delivery & Architectural integrity
    8. 8. Quick & Dirty ≠ Agile (3)
    9. 9. BI requirements Information products trigger change requests: new data ⇒ insights ⇒ new requirements Gerald M. (Jerry) Weinberg: “Without stable requirements, development can’t stabilize, either”
    10. 10. BI: means and ends uncertaintyMeans uncertainty Ends uncertainty How do we get there?  Where are we going to? Lack of “design patterns”  Requirements are difficult Data integration fraught to pin down with data quality issues  Diverse end-user groups Lack of Master Data  Ambiguous business Management case(s) Lack of Meta Data  Scope is unclear No agreement on how to  Data warehouses are conform dimensions never “done” www.xlntconsulting.com 10
    11. 11. Waterfall ⇔ Agile Waterfall/ Traditional Agile Requirements Resources Date Fixed Value Driven Plan Driven Estimated Resources Date Requirements Agile fixes the date and resources and varies the scope source: Dean Leffingwell (2011) www.xlntconsulting.com 11
    12. 12. Weinberg on Quality “If quality isn’t an objective (if the software doesn’t have to work), you can satisfy any other constraint (e.g.: budget, time, etc.)” Gerald M. (Jerry) Weinberg www.xlntconsulting.com 12
    13. 13. Concurrent development (1) Waterfall: you can avoid mistakes/rework by getting good requirements upfront The most costly mistakes arise from forgetting important elements early on Detailed planning (BDUF) requires:  early (ill informed) decisions  uses more time  leading to less tangible products to re s o lve ambiguity www.xlntconsulting.com ⇒ vicious cycle 13
    14. 14. Concurrent development (2) Agile: decide at “last responsible moment”  decisions that haven’t been made, don’t ever need to be reverted No “free lunch” – deferring decisions requires:  anticipating likely change  coordination/collaboration within team  close contact with customers www.xlntconsulting.com 14
    15. 15. Inmon ⇔ Kimball (1) 3-tiered 2-tiered
    16. 16. Inmon ⇔ Kimball (2)Problems with Inmon Problems with Kimball Uncovering the ‘correct’  Smallest unit of delivery 3NF model requires is a Star, and incremental scarce business expertise growth adds prohibitive Unclear where 3NF overhead model boundaries begin  Dimensional structure is and end very rigid → not Model redesigns trigger a conducive to expansion cascading nightmare of or change parent-child key updates  Conforming dimensions is hard, especially without access to data www.xlntconsulting.com 16
    17. 17. 3NF ⇔ Dimensional (1) www.xlntconsulting.com 17
    18. 18. 3NF ⇔ Dimensional (2) this problem gets (much!) worse with multiple parent-child levels see: Kimball design tip # 149 http://www.kimballgroup.com/2012/10/02/d esign-tip-149-facing-the-re-keying-crisis/ www.xlntconsulting.com 18
    19. 19. Hyper normalized model business keys, context attributes (history), and relations, all have their own tables appending “Supplier data” to the model (or any other new source), is guaranteed to be contained as a “local” problem (=extension) in the data model because business keys, context attributes (history), and relations www.xlntconsulting.com 19 all have their own tables
    20. 20. 3-tiered DWH architecture Metadata Legacy Datamart 1 OLTP Datamart 2 Data Warehouse ERP ETL Staging Business Area Intelligence LOG Datamart n Applications files ODS External hyper 3 www.xlntconsulting.com NF normalized 20 dimensional
    21. 21. Horses for courses 3NF  quickly & accurately capture transaction data  easy to get data in Hyper normalized  integrate historical data  capture all data, all the time Dimensional  present & analyze data  easy to get data out www.xlntconsulting.com 21
    22. 22. Backroom ⇔ Frontroom Back room Metadata Front room Data Warehouse Architecture Legacy Business Intelligence Architecture Datamart 1 OLTP Datamart 2 Data Warehouse ERP ETL Staging Business Area Intelligence LOG Datamart n Applications files ODS External www.xlntconsulting.com 22
    23. 23. Divide & Conquer “Break down” semantic gap from back- to front room Offer a range of data services:  Source data “as is”  Source data that have undergone cleansing  Dimensional models  Full-fledge BI applications Allow business to set priorities!
    24. 24. Why data virtualization? Operational BI calls for real-time data Integrate heterogeneous sources, at least “in the eye of the beholder” Data virtualization layer hides complexity about underlying applications  & enables sharing of meta data Data virtualization enables federation, so you can delay (definitive) modeling, yet make data available early
    25. 25. Conclusion Big Data are here to stay (and lets hope the hype passes soon) Data provide a source of sustainable competitive advantage Speed and volume prohibit (wholesale) copying: virtualization is the way forward Agile BI enables business alignment, and gives us a “sporting chance” to keep up
    26. 26. Conclusion Big Data are here to stay (and lets hope the hype passes soon) Data provide a source of sustainable competitive advantage Speed and volume prohibit (wholesale) copying: virtualization is the way forward Agile BI enables business alignment, and gives us a “sporting chance” to keep up
    27. 27. Conclusion Big Data are here to stay (and lets hope the hype passes soon) Data provide a source of sustainable competitive advantage Speed and volume prohibit (wholesale) copying: virtualization is the way forward Agile BI enables business alignment, and gives us a “sporting chance” to keep up
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×