Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Agile BI Development Through Automation


Published on

How can code life cycle automation satisfy the growing demands in modern enterprise business intelligence?

Whilst an agile approach to BI development is useful for delivering value in general, the use of advanced automation techniques can also save significant resources, prevent production errors, and shorten time to market.

Gentlemen from Data To Value, Manta Tools, Volkswagen and M&G investments presented and discussed different approaches to agile BI development. Take a look!

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Agile BI Development Through Automation

  1. 1. Tomas Kratky CEO Manta Tools Nigel Higgs Co-founder Data To Value
  2. 2. Agenda 03 12 30 44 68
  3. 3. Introduction to services
  4. 4. We help organisations get more value from their Data Architecture 5 Core practice areas covering both business and architecture aspects of data managment Service delivery through: Onsite consulting Onsite / Offsite Managed Services Expert users of technology accelerators for bridging technology & business Data gap
  5. 5. Lean Data Management Focus on reducing waste & minimising TCO Unification of unstructured information / knowledge management & structured data management Key mantra is minimising time spent on building solutions customers do not want Lean Information Managemen t Shorter iterations Prototyping & Minimum Viable Products Build- Measure- Learn cycle Early adopters Cross functional teams Actionable metrics
  6. 6. Unique Value Proposition Experience & expertise Founders have over 40 years experience working with data Skilled in defining data strategy and implementing data architecture, governance and analytics solutions Focussed on delivering business value Align data strategy with client’s strategic goals Work packages based on business case and ROI Lean, agile & iterative approach Hybrid consultancy model to scale to meet demand Partner with innovative vendors of data tools software
  7. 7. We have a number of industry partnerships that allow us to hit the ground running We also use industry leading platforms such as AWS for hosting and Tableau for Data Visualisation Our focus is on providing customers with the most appropriate tooling to continue to make progress after initial projects have completed Tooling & Partners Data to Value industry partners & platforms:
  8. 8. Lean Approach – Iterative Process Maturity benchmarking Data Profiling & Data Discovery Harvest key metadata (apps, lineage, processes etc.) Test rules & capture metrics Generate risk & cost metrics Capture quality, governance & modelling notes Review issues using visualisations & dashboards Prototype data solutions Implement practical
  9. 9. Integrated Approach data quality & governance metrics data models, glossaries & dictionaries disparate data & metadata data profiling & metadata discovery ontologies controlled vocabularies
  10. 10. Typical Outputs DQ Issue lists & KPIs to guide decision making Powerful, interactive visualisations Models, Knowledge Graphs & Glossaries to understand what data assets you have Dashboards articulating the data quality issues that are holding you back Clean, actionable & well structured data in a variety of formats Ready to use Prototypes & POCs
  11. 11. Passionate, Innovative, Lean Lean Information Management specialists Data to Value Ltd. 2nd Floor Elizabeth House, Waterloo, London SE1 7NQ United Kingdom T +44 (0) 208 278 7351
  12. 12. • • • • • •
  13. 13. • • •
  14. 14. • • • •
  15. 15. • • • •
  16. 16. Enterprise vs Business Driven BI Raju Sonawane
  17. 17. About Me  Twenty Six years’ experience in Business Intelligence, enterprise architecture, strategy/roadmap, design, development and project/people management.  Roles played - Head of Business Intelligence, Data Architect, BI Architect, Solution Architect, Agile Scrum Master, Project Manager and onsite/offshore Business Development Manager  Domain/Industry - Fund/Investment Management, Lloyds of London Specialist Insurance/ Reinsurance, Life Insurance and Consultancy The content in this presentation is my opinion/view of BI. My current/previous employers may have different views.
  18. 18. BI Maturity in my view
  19. 19. Enterprise BI
  20. 20. Ways to deliver it
  21. 21. Strengths and Weaknesses
  22. 22. The side effect – Homegrown UDAs
  23. 23. Business Driven BI
  24. 24. Ways to deliver it Strategy •Define the Enterprise BI Strategy •Set the BI products roadmap Enable the Platform •Open the platform to the users •Empower the users with self-service Rapid Prototypes •Promote “BI as a Service” •Identify the high value low size use cases •Build the rapid prototypes along with the users Leverage User-driven BI •Govern the user-driven BI •Leverage the popular user-driven BI to build Enterprise BI
  25. 25. Ways to deliver it Demo the existing BI capabilities Conduct Workshops for the new use cases Deliver the rapid prototypes with the users
  26. 26. Ways to deliver it
  27. 27. Strengths and Weaknesses
  28. 28. BI Strategy
  29. 29. Finally… Thank You
  30. 30. Oliver Cramer The Model is the Foundation for Data Warehouse Automation Agile BI Development Through Automation London
  31. 31. DWH42 Agenda • About me • Understanding • The gap • The model • Data Warehouse Automation
  32. 32. DWH42 About me • Data Warehouse Architect • 13 years working in the Business Intelligence area • Since 2003 working with elementary building blocks for the Data Warehouse • Blog Data Warehouse Automation • Interested in the exchange of knowledge about Core Data Warehouse modeling styles
  33. 33. DWH42 About me • TDWI Europe Fellow • ANCHOR CERTIFIED MODELER Version 2014 • Certified Data Vault 2.0 Practitioner • Coautor of „Neue Wege in der Datenmodellierung - Data Vault heißt die moderne Antwort“ in BI-Spektrum 03-2014 • Member of the Boulder BI Brain Trust • Member of the BI-Podium Advisory Board Germany • Responsible editor of the TDWI Germany Online Special „Data Vault“ • Organizer Data Vault Modeling and Certification, Hannover with Genesee Academy (CDVDM course)
  34. 34. DWH42 Agenda • About me • Understanding • The gap • The model • Data Warehouse Automation
  35. 35. DWH42 The maturity path of understanding • Multiple perspectives on the facts = Data Warehouse as an enabler to make your own picture of the world from existing data and information! • One version of the facts = Data Warehouse as a recording device • One version of the truth = Data Warehouse delivers the truth
  36. 36. DWH42 Data: consistency vs. availability There is a fundamental choice to be made when data is to be 'processed': • a choice between consistency vs. availability or • a choice between work upstream vs. work downstream or • a choice between a sustainable (long term) view vs. an opportunistic (short term) view on data Ronald Damhof
  37. 37. DWH42 The confusion solution Lars Rönnback: "When working with information, confusion is sometimes unavoidable. To be more precise, when the process of identification cannot give unambiguous results, such confusion arises. ... Push that problem into the future, to solve it when you find the missing pieces, while still retaining analytic capabilities. Simply store all the possible outcomes in advance, with different reliabilities, or store the most likely scenario and correct it later if it was wrong.
  38. 38. DWH42 Main model requirements • The model must be capable to absorb multiple perspectives on the facts! • The model must be capable of corrections!
  39. 39. DWH42 Our problem -> rendering knowledge Dave Snowden: 7 Principles of Knowledge Management / Rendering Knowledge: 1. Knowledge can only be volunteered, it cannot be conscripted. 2. We only know what we know when we need to know it. 3. In the context of real need few people will withhold their knowledge. 4. Everything is fragmented. 5. Tolerated failure imprints learning better than success. 6. The way we know things is not the way we report we know things. 7. We always know more than we can say, and we always say more than we can write down.
  40. 40. DWH42 Agenda • About me • Understanding • The gap • The model • Data Warehouse Automation
  41. 41. DWH42 The gap The big gap between modelers and business people is the language we speak! The modelers mantra: • We have to close the gap! • They will never close the gap! • They will not move in our direction!
  42. 42. DWH42 The logic • We aspire to be logical modelers, to create the best logical model! • Are the business people logical? Are they like Spock from the Starship Enterprise? Are they from the planet Vulcan? • No, they are humans from the planet earth like we are!
  43. 43. DWH42 The advancement • The model must have a fully communication orientation (in this case business speech) (is that logical modeling?) • For this reason the model must support homonyms and synonyms! • A synonym is a word or phrase that means exactly or nearly the same as another word or phrase in the same language. • In linguistics, a homonym is one of a group of words that share the same pronunciation but have different meanings, whether spelled the same or not.
  44. 44. DWH42 Fully communication orientation -> Business model From Quipu: • This business model does not normally exist in any source system: it must be developed in close cooperation with the business to reflect the terms and definition of the data that the business chooses to work with. It identifies the business keys that identify the various business entities and their inter- relations. It also specifies all relevant attributes and facts related to these business entities that are required for management reporting, (predictive) analysis, etc.
  45. 45. DWH42 Agenda • About me • Understanding • The gap • The model • Data Warehouse Automation
  46. 46. DWH42 The model Model requirements: • It must have integration points. • It must support identification. • It must support relationships / Unit of Work relationships! • It must support dynamic relationships. • It must support storing attributes from different origins / integration of attributes is not necessary! • It must categorize attributes for identification. • It must be a model with historization capabilities. And history of history?
  47. 47. DWH42 The model Model requirements: • Support of data provenance! Data provenance refers to the ability to trace and verify the creation of data, how it has been used or moved among different databases, as well as altered throughout its lifecycle. • It must follow standards. • It must follow naming conventions. • It must follow patterns.
  48. 48. DWH42 The model Model requirements: • It must be scalable. • It must be readable and understandable! • It must be searchable. Crawler! • It must be partition able. • It must be extendable. • The possibility to model extensions without destruction of current entities! • It must be version able.
  49. 49. DWH42 The model Model requirements: • It must support the separation of concerns! • It must have a raw data area. • It must have a integration area. • It must have a rule area. • It must have a area for sensible data. • It must have a business area. • It must support temporal and business perspectives.
  50. 50. DWH42 Agenda • About me • Understanding • The gap • The model • Data Warehouse Automation
  51. 51. DWH42 Data Warehouse Automation The big picture All these details might make it hard to understand how this has anything to do with automation of Data Warehouse. • The only two steps that can’t get automated are the information modeling process and the semantic mapping exercise. • Is that statement subject to change? • Today the rest, before applying of rules, is the domain of data warehouse automation! • And what can be done ...!
  52. 52. DWH42 Data Warehouse Automation Baseline is that the model must separate keys/identifiers, relationships and attributes / group of attributes for Data Warehouse Automation. It must be fully communication oriented, so we can close the gap to the business. In the end we can focus on asking better questions. This is the next generation of Data Warehouse Automation.
  53. 53. DWH42 End Thanks for the attention!
  54. 54. • Models • Business Glossary • Relations • … Implementation • Marts • Reports • ETLs • … • Enhancement • Consolidation • Restructuralization
  55. 55. • • • •
  56. 56. • • • •
  57. 57. Panel discussion
  58. 58. Connected Data London 2016 The leading conference bringing together the Linked and Graph Data communities 12th of July – Central London @Connected_Data
  59. 59. Connected Data London MeetUp Join us at our informal MeetUp event. Listen to short talks delivered by Connected dData experts and share your ideas with like minded Connected Data fans! 7th of June – Central London @Connected_Data
  60. 60. Thank you for coming! Please fill out our survey before leaving @DataToValue @Manta_tools