Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Tom Breur, XLNT - Agile BI And Data Virtualization - BI Symposium 2012

1,474 views

Published on

  • DOWNLOAD FULL BOOKS, INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/yxufevpm } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/yxufevpm } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/yxufevpm } ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/yxufevpm } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/yxufevpm } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/yxufevpm } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Tom Breur, XLNT - Agile BI And Data Virtualization - BI Symposium 2012

  1. 1. Agile BI& Data Virtualization Tom Breur tombreur@xlntconsulting.com BI & IM Symposium Bussum, 26 November 2012
  2. 2. It’s a stretch… Volumes of data are growing (fast)! Variety of sources keeps expanding:  Social media, RFID, log-files, GPS, etc. Business users need their data (much) sooner:  monthly ⇒ weekly ⇒ daily ⇒ intra-day BI in support of operational processes, calls for (near) real-time data
  3. 3. Why go “Agile”? (1) BI projects fail too often, or don’t live up to expectations Increasingly, BI development takes place alongside (instead of a fte r) application engineering
  4. 4. Why go “Agile”? (2) Winston Royce (1970): “In my experience, the simpler model … [as pictured below] has never worked on large software development efforts” Analysis Design Development Test Release [Royce subsequently went on to describe an enhanced model, whichincluded building a prototype first and then using the prototype plus feedback between phases to build a final deployment] www.xlntconsulting.com 4
  5. 5. History of development methods Jackson (1975) “Principles of Program Design” Martin (1991) “Rapid Application Development” Anderson (2010) “Kanban” Boehm (1986) “A Spiral model of Software Development and Enhancement” Beck (2000) “Extreme Programming Explained” Brooks (1975) “The Mythical Man Month” Poppendieck (2003) “Lean Software Development”1960 1970 1980 1990 2000 2010 1994: DSDM Consortium launchedRoyce (1970) “Managing the Development of Large Software Systems” Cockburn (2004) “Crystal Clear” 1997: term eXtreme Programming (XP) ‘invented’ 1996: Scrum ‘invented’ 2001: term ‘Agile’ adopted From “code and fix” to more structured, methodical approaches to software development Unstructured Prescriptive methods Structured methods www.xlntconsulting.com 5
  6. 6. Quick & Dirty ≠ Agile (1) www.agilemanifesto.org (principle #1): “Our highest priority is to satisfy the customer through early and continuous delivery of valuable software” [emphasis added] Creating “technical debt” stands squarely in the way of continuous delivery, and maintaining a so-called “sustainable pace”:
  7. 7. Quick & Dirty ≠ Agile (2) Top-down project management (e.g.: Scrum) & Bottom-up software engineering (e.g.: Extreme Programming - XP) ⇓ Expedited delivery & Architectural integrity
  8. 8. Quick & Dirty ≠ Agile (3)
  9. 9. BI requirements Information products trigger change requests: new data ⇒ insights ⇒ new requirements Gerald M. (Jerry) Weinberg: “Without stable requirements, development can’t stabilize, either”
  10. 10. BI: means and ends uncertaintyMeans uncertainty Ends uncertainty How do we get there?  Where are we going to? Lack of “design patterns”  Requirements are difficult Data integration fraught to pin down with data quality issues  Diverse end-user groups Lack of Master Data  Ambiguous business Management case(s) Lack of Meta Data  Scope is unclear No agreement on how to  Data warehouses are conform dimensions never “done” www.xlntconsulting.com 10
  11. 11. Waterfall ⇔ Agile Waterfall/ Traditional Agile Requirements Resources Date Fixed Value Driven Plan Driven Estimated Resources Date Requirements Agile fixes the date and resources and varies the scope source: Dean Leffingwell (2011) www.xlntconsulting.com 11
  12. 12. Weinberg on Quality “If quality isn’t an objective (if the software doesn’t have to work), you can satisfy any other constraint (e.g.: budget, time, etc.)” Gerald M. (Jerry) Weinberg www.xlntconsulting.com 12
  13. 13. Concurrent development (1) Waterfall: you can avoid mistakes/rework by getting good requirements upfront The most costly mistakes arise from forgetting important elements early on Detailed planning (BDUF) requires:  early (ill informed) decisions  uses more time  leading to less tangible products to re s o lve ambiguity www.xlntconsulting.com ⇒ vicious cycle 13
  14. 14. Concurrent development (2) Agile: decide at “last responsible moment”  decisions that haven’t been made, don’t ever need to be reverted No “free lunch” – deferring decisions requires:  anticipating likely change  coordination/collaboration within team  close contact with customers www.xlntconsulting.com 14
  15. 15. Inmon ⇔ Kimball (1) 3-tiered 2-tiered
  16. 16. Inmon ⇔ Kimball (2)Problems with Inmon Problems with Kimball Uncovering the ‘correct’  Smallest unit of delivery 3NF model requires is a Star, and incremental scarce business expertise growth adds prohibitive Unclear where 3NF overhead model boundaries begin  Dimensional structure is and end very rigid → not Model redesigns trigger a conducive to expansion cascading nightmare of or change parent-child key updates  Conforming dimensions is hard, especially without access to data www.xlntconsulting.com 16
  17. 17. 3NF ⇔ Dimensional (1) www.xlntconsulting.com 17
  18. 18. 3NF ⇔ Dimensional (2) this problem gets (much!) worse with multiple parent-child levels see: Kimball design tip # 149 http://www.kimballgroup.com/2012/10/02/d esign-tip-149-facing-the-re-keying-crisis/ www.xlntconsulting.com 18
  19. 19. Hyper normalized model business keys, context attributes (history), and relations, all have their own tables appending “Supplier data” to the model (or any other new source), is guaranteed to be contained as a “local” problem (=extension) in the data model because business keys, context attributes (history), and relations www.xlntconsulting.com 19 all have their own tables
  20. 20. 3-tiered DWH architecture Metadata Legacy Datamart 1 OLTP Datamart 2 Data Warehouse ERP ETL Staging Business Area Intelligence LOG Datamart n Applications files ODS External hyper 3 www.xlntconsulting.com NF normalized 20 dimensional
  21. 21. Horses for courses 3NF  quickly & accurately capture transaction data  easy to get data in Hyper normalized  integrate historical data  capture all data, all the time Dimensional  present & analyze data  easy to get data out www.xlntconsulting.com 21
  22. 22. Backroom ⇔ Frontroom Back room Metadata Front room Data Warehouse Architecture Legacy Business Intelligence Architecture Datamart 1 OLTP Datamart 2 Data Warehouse ERP ETL Staging Business Area Intelligence LOG Datamart n Applications files ODS External www.xlntconsulting.com 22
  23. 23. Divide & Conquer “Break down” semantic gap from back- to front room Offer a range of data services:  Source data “as is”  Source data that have undergone cleansing  Dimensional models  Full-fledge BI applications Allow business to set priorities!
  24. 24. Why data virtualization? Operational BI calls for real-time data Integrate heterogeneous sources, at least “in the eye of the beholder” Data virtualization layer hides complexity about underlying applications  & enables sharing of meta data Data virtualization enables federation, so you can delay (definitive) modeling, yet make data available early
  25. 25. Conclusion Big Data are here to stay (and lets hope the hype passes soon) Data provide a source of sustainable competitive advantage Speed and volume prohibit (wholesale) copying: virtualization is the way forward Agile BI enables business alignment, and gives us a “sporting chance” to keep up
  26. 26. Conclusion Big Data are here to stay (and lets hope the hype passes soon) Data provide a source of sustainable competitive advantage Speed and volume prohibit (wholesale) copying: virtualization is the way forward Agile BI enables business alignment, and gives us a “sporting chance” to keep up
  27. 27. Conclusion Big Data are here to stay (and lets hope the hype passes soon) Data provide a source of sustainable competitive advantage Speed and volume prohibit (wholesale) copying: virtualization is the way forward Agile BI enables business alignment, and gives us a “sporting chance” to keep up

×