Your SlideShare is downloading. ×
Tom Breur, XLNT - Agile BI And Data Virtualization - BI Symposium 2012
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Saving this for later?

Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime - even offline.

Text the download link to your phone

Standard text messaging rates apply

Tom Breur, XLNT - Agile BI And Data Virtualization - BI Symposium 2012

595
views

Published on


0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
595
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
50
Comments
0
Likes
1
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide
  • Agile BI & Data Virtualization 26-NOV-2012 DWH Automation conference
  • Agile BI - How it works and ho to do it 6-NOV-2012 www.xlntconsulting.com
  • Agile BI - How it works and ho to do it 6-NOV-2012 DWH Automation conference
  • Agile BI - How it works and ho to do it 6-NOV-2012 DWH Automation conference
  • Agile BI & Data Virtualization 26-NOV-2012 www.xlntconsulting.com
  • Agile BI & Data Virtualization 26-NOV-2012 www.xlntconsulting.com
  • Agile BI & Data Virtualization 26-NOV-2012 www.xlntconsulting.com
  • Agile BI & Data Virtualization 26-NOV-2012 www.xlntconsulting.com
  • Agile BI & Data Virtualization 26-NOV-2012 www.xlntconsulting.com
  • Transcript

    • 1. Agile BI& Data Virtualization Tom Breur tombreur@xlntconsulting.com BI & IM Symposium Bussum, 26 November 2012
    • 2. It’s a stretch… Volumes of data are growing (fast)! Variety of sources keeps expanding:  Social media, RFID, log-files, GPS, etc. Business users need their data (much) sooner:  monthly ⇒ weekly ⇒ daily ⇒ intra-day BI in support of operational processes, calls for (near) real-time data
    • 3. Why go “Agile”? (1) BI projects fail too often, or don’t live up to expectations Increasingly, BI development takes place alongside (instead of a fte r) application engineering
    • 4. Why go “Agile”? (2) Winston Royce (1970): “In my experience, the simpler model … [as pictured below] has never worked on large software development efforts” Analysis Design Development Test Release [Royce subsequently went on to describe an enhanced model, whichincluded building a prototype first and then using the prototype plus feedback between phases to build a final deployment] www.xlntconsulting.com 4
    • 5. History of development methods Jackson (1975) “Principles of Program Design” Martin (1991) “Rapid Application Development” Anderson (2010) “Kanban” Boehm (1986) “A Spiral model of Software Development and Enhancement” Beck (2000) “Extreme Programming Explained” Brooks (1975) “The Mythical Man Month” Poppendieck (2003) “Lean Software Development”1960 1970 1980 1990 2000 2010 1994: DSDM Consortium launchedRoyce (1970) “Managing the Development of Large Software Systems” Cockburn (2004) “Crystal Clear” 1997: term eXtreme Programming (XP) ‘invented’ 1996: Scrum ‘invented’ 2001: term ‘Agile’ adopted From “code and fix” to more structured, methodical approaches to software development Unstructured Prescriptive methods Structured methods www.xlntconsulting.com 5
    • 6. Quick & Dirty ≠ Agile (1) www.agilemanifesto.org (principle #1): “Our highest priority is to satisfy the customer through early and continuous delivery of valuable software” [emphasis added] Creating “technical debt” stands squarely in the way of continuous delivery, and maintaining a so-called “sustainable pace”:
    • 7. Quick & Dirty ≠ Agile (2) Top-down project management (e.g.: Scrum) & Bottom-up software engineering (e.g.: Extreme Programming - XP) ⇓ Expedited delivery & Architectural integrity
    • 8. Quick & Dirty ≠ Agile (3)
    • 9. BI requirements Information products trigger change requests: new data ⇒ insights ⇒ new requirements Gerald M. (Jerry) Weinberg: “Without stable requirements, development can’t stabilize, either”
    • 10. BI: means and ends uncertaintyMeans uncertainty Ends uncertainty How do we get there?  Where are we going to? Lack of “design patterns”  Requirements are difficult Data integration fraught to pin down with data quality issues  Diverse end-user groups Lack of Master Data  Ambiguous business Management case(s) Lack of Meta Data  Scope is unclear No agreement on how to  Data warehouses are conform dimensions never “done” www.xlntconsulting.com 10
    • 11. Waterfall ⇔ Agile Waterfall/ Traditional Agile Requirements Resources Date Fixed Value Driven Plan Driven Estimated Resources Date Requirements Agile fixes the date and resources and varies the scope source: Dean Leffingwell (2011) www.xlntconsulting.com 11
    • 12. Weinberg on Quality “If quality isn’t an objective (if the software doesn’t have to work), you can satisfy any other constraint (e.g.: budget, time, etc.)” Gerald M. (Jerry) Weinberg www.xlntconsulting.com 12
    • 13. Concurrent development (1) Waterfall: you can avoid mistakes/rework by getting good requirements upfront The most costly mistakes arise from forgetting important elements early on Detailed planning (BDUF) requires:  early (ill informed) decisions  uses more time  leading to less tangible products to re s o lve ambiguity www.xlntconsulting.com ⇒ vicious cycle 13
    • 14. Concurrent development (2) Agile: decide at “last responsible moment”  decisions that haven’t been made, don’t ever need to be reverted No “free lunch” – deferring decisions requires:  anticipating likely change  coordination/collaboration within team  close contact with customers www.xlntconsulting.com 14
    • 15. Inmon ⇔ Kimball (1) 3-tiered 2-tiered
    • 16. Inmon ⇔ Kimball (2)Problems with Inmon Problems with Kimball Uncovering the ‘correct’  Smallest unit of delivery 3NF model requires is a Star, and incremental scarce business expertise growth adds prohibitive Unclear where 3NF overhead model boundaries begin  Dimensional structure is and end very rigid → not Model redesigns trigger a conducive to expansion cascading nightmare of or change parent-child key updates  Conforming dimensions is hard, especially without access to data www.xlntconsulting.com 16
    • 17. 3NF ⇔ Dimensional (1) www.xlntconsulting.com 17
    • 18. 3NF ⇔ Dimensional (2) this problem gets (much!) worse with multiple parent-child levels see: Kimball design tip # 149 http://www.kimballgroup.com/2012/10/02/d esign-tip-149-facing-the-re-keying-crisis/ www.xlntconsulting.com 18
    • 19. Hyper normalized model business keys, context attributes (history), and relations, all have their own tables appending “Supplier data” to the model (or any other new source), is guaranteed to be contained as a “local” problem (=extension) in the data model because business keys, context attributes (history), and relations www.xlntconsulting.com 19 all have their own tables
    • 20. 3-tiered DWH architecture Metadata Legacy Datamart 1 OLTP Datamart 2 Data Warehouse ERP ETL Staging Business Area Intelligence LOG Datamart n Applications files ODS External hyper 3 www.xlntconsulting.com NF normalized 20 dimensional
    • 21. Horses for courses 3NF  quickly & accurately capture transaction data  easy to get data in Hyper normalized  integrate historical data  capture all data, all the time Dimensional  present & analyze data  easy to get data out www.xlntconsulting.com 21
    • 22. Backroom ⇔ Frontroom Back room Metadata Front room Data Warehouse Architecture Legacy Business Intelligence Architecture Datamart 1 OLTP Datamart 2 Data Warehouse ERP ETL Staging Business Area Intelligence LOG Datamart n Applications files ODS External www.xlntconsulting.com 22
    • 23. Divide & Conquer “Break down” semantic gap from back- to front room Offer a range of data services:  Source data “as is”  Source data that have undergone cleansing  Dimensional models  Full-fledge BI applications Allow business to set priorities!
    • 24. Why data virtualization? Operational BI calls for real-time data Integrate heterogeneous sources, at least “in the eye of the beholder” Data virtualization layer hides complexity about underlying applications  & enables sharing of meta data Data virtualization enables federation, so you can delay (definitive) modeling, yet make data available early
    • 25. Conclusion Big Data are here to stay (and lets hope the hype passes soon) Data provide a source of sustainable competitive advantage Speed and volume prohibit (wholesale) copying: virtualization is the way forward Agile BI enables business alignment, and gives us a “sporting chance” to keep up
    • 26. Conclusion Big Data are here to stay (and lets hope the hype passes soon) Data provide a source of sustainable competitive advantage Speed and volume prohibit (wholesale) copying: virtualization is the way forward Agile BI enables business alignment, and gives us a “sporting chance” to keep up
    • 27. Conclusion Big Data are here to stay (and lets hope the hype passes soon) Data provide a source of sustainable competitive advantage Speed and volume prohibit (wholesale) copying: virtualization is the way forward Agile BI enables business alignment, and gives us a “sporting chance” to keep up