Why ask why? Try agile BI!
•Download as PPTX, PDF•
0 likes•1,339 views
Sara Handel's presentation from AgileDC on October 21, 2014.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Why ask why? Try agile BI!
Similar to Why ask why? Try agile BI! (20)
More from Excella
More from Excella (20)
Recently uploaded
Recently uploaded (20)
Why ask why? Try agile BI!
- 1. VOTE on the Biggest BI Challenges Welcome! As you’re getting settled, take a minute to put a dot on the board by what you think are the three biggest BI challenges.
- 2. Why Ask Why Try Agile BI November 5, 2014
- 3. Agenda • Agile BI Definition • Assess Your Current State • Then… – Select an Agile Methodology – Have a Kickoff – Inspect and Adapt
- 4. BI BI encompasses all aspects of a system needed to produce meaningful information to drive data driven decision making – Data Processing (Cleansing, Transforming, Loading) – Data Architecture and Warehousing – Data Analysis and Visualization Tools
- 5. Agile
- 6. Agile BI Applying an Agile mindset to business intelligence • Using an iterative, incremental, evolutionary approach • Focusing on value-driven development • Delivering production quality applications • Using barely sufficient processes • Automating everything • Collaborating with the customer • Encouraging self-organizing and self managing teams
- 7. Why Agile? The people who need the data see the data You need different data? Sure! The data source changed? We’re on it! Business Intelligence, Source: http://www.versionone.com/Agile101/Agile-Software-Development-Benefits/ enough said Fail fast
- 8. Agile BI Maturity Model Team Roles / Skill Sets Technical Architecture Engineering Practices *All team members can independently complete any task from database design to report creation *It’s not about getting your job done it’s about getting the job done *Increased collaboration Example: ETL developers work with data modelers to come up with a database design that balances the tradeoffs between reporting and loading *Decreased formality in interactions across skill sets *Collaboration among people with the same skill set - Example: data modelers work with other data modelers *Official transitions and likely disagreement across skill sets - Example: ETL developers are given source to target mappings when the data modelers complete the database design and are upset that the design is hard to load *Clear understanding of data’s business value *Clear understanding of the purpose for each component of the technical architecture *Active effort to clarify understanding of data’s business value *Streamlined architecture where possible *Process to deprecate unused components *Numerous (possibly) redundant layers (staging, ODS, EDW, data marts, etc.) *Inclusion of data with no clear business value *Lingering tables, reports, ETL scripts, with no known purpose *End-to-end use of optimal engineering practices *Team self-enforces usage through criteria for completing work *Some configuration management (SQL scripts to create all db objects are under CM, but not ETL and report information) *Some automation is in place (perhaps to promote new objects or code to another environment or to test ETL) Level 1 Level 2 Level 3
- 9. Assess Your Current State • How well is your team setup for collaboration and change?
- 10. Agile BI Maturity Model Team Roles / Skill Sets It’s not about getting your job done it’s about getting the job done Level 3 Level 2 Level 1 Collaboration Formality ETL Data Reporting Modelers DBAs etc. https://www.castlellc.com/collaboration.aspx
- 11. Agile BI Maturity Model Team Roles / Skill Sets Support self organized culture - Let the team define their own success criteria - Avoid saying HOW things must be done Fill skill set gaps with external training, cross training, lunch and learns and more Create a dedicated team with skills needed to get data into the hands of end users to make decisions Level 3 Level 2 Level 1
- 12. Assess Your Current State • How well is your team setup for collaboration and change?
- 13. Assess Your Current State • What is your current technical architecture? What aspects present the biggest challenges to incremental evolution and change?
- 14. Change Is… • Grain of fact table • New type 2 attribute • Change from type 1 to type 2 • Multi-purpose column or table • Redundant data • Tables with too many columns or rows • “Smart” columns • Complex ETL objects • Large SQL modules • Unconformed Dimensions • Indiscriminate use of materialized views • Underutilization of materialized views • Overreliance on documentation Avoidable Inevitable
- 15. Agile BI Maturity Model Technical Architecture Level 3 Level 2 Level 1 Purpose! Value! Purpose? Value? Purpose?? Value??
- 16. Agile BI Maturity Model Technical Architecture Keep it up! Don’t let complexity creep in. *Create a central repository *Get rid of things that are no longer being used * Identify redundancy * Combine or streamline things where possible Level 3 Level 2 Level 1
- 17. Assess Your Current State • What is your current technical architecture? What aspects present the biggest challenges to incremental evolution and change?
- 18. Assess Your Current State • Do you follow technical practices that can enable agility?
- 19. Agile BI Maturity Model Engineering Practices Level 3 Level 2 Level 1 End-to-end use of optimal engineering practices • Some configuration management • Some automation • No central location for system building blocks • Manual push between environments
- 20. Agile BI Maturity Model Engineering Practices *Hold yourselves accountable for maintaining high standards for new efforts *Reduce technical debt each iteration *Start creating automated tests *Start putting files into a configuration management system *Work out the kinks of your deployment process Level 3 Level 2 Level 1
- 21. Assess Your Current State • Do you follow technical practices that can enable agility?
- 22. Agile BI Maturity Model Team Roles / Skill Sets Technical Architecture Engineering Practices *All team members can independently complete any task from database design to report creation *It’s not about getting your job done it’s about getting the job done *Increased collaboration Example: ETL developers work with data modelers to come up with a database design that balances the tradeoffs between reporting and loading *Decreased formality in interactions across skill sets *Collaboration among people with the same skill set - Example: data modelers work with other data modelers *Official transitions and likely disagreement across skill sets - Example: ETL developers are given source to target mappings when the data modelers complete the database design and are upset that the design is hard to load *Clear understanding of data’s business value *Clear understanding of the purpose for each component of the technical architecture *Active effort to clarify understanding of data’s business value *Streamlined architecture where possible *Process to deprecate unused components *Numerous (possibly) redundant layers (staging, ODS, EDW, data marts, etc.) *Inclusion of data with no clear business value *Lingering tables, reports, ETL scripts, with no known purpose *End-to-end use of optimal engineering practices *Team self-enforces usage through criteria for completing work *Some configuration management (SQL scripts to create all db objects are under CM, but not ETL and report information) *Some automation is in place (perhaps to promote new objects or code to another environment or to test ETL) *Building blocks of the system (db create scripts, ETL packages, report files, etc.) are not maintained in any central location nor are they under configuration management *Files are manually copied from one environment to another Level 1 Level 2 Level 3
- 23. Select an Agile Methodology Scrum Kanban RU P XP BDD And so on… Scrum
- 24. Have a Kick Off
- 26. Contact Information Sara Handel sara.handel@excella.com Agile BI Training – December 11, 2014 Check out Wyn Van Devanter next A Thin Automation Framework for Manageable Automated Acceptance Testing
Editor's Notes
- Too hard/takes too long to add data Not enough people to do BI work (lack of technical expertise) Tons of data, reports, etc., but they aren’t being used Enormous scope, no access to data while the project is in progress Hard to change existing BI system to support new requirements End users do not trust the data Management does not understand the value of data
- Agile development is a business-value-driven approach that emphasizes the continuous delivery of high-priority and high-quality capabilities to business customers. It involves a high degree of collaboration with end users and stakeholders, and is open to change for the sake of optimum project outcomes.
- Who is on your team? What are their skills, and are they able to perform work cross functionally? For example, can your data architects also write ETL code? Can your ETL programmers also design database tables?
- How is work getting done now? Who is doing what? How do you complete a full “slice” of functionality needed by end users? LEVEL1 **Official transitions and likely disagreement across skill sets - Example: ETL developers are given source to target mappings when the data modelers complete the database design and are upset that the design is hard to load LEVEL 2 *Increased collaboration Example: ETL developers work with data modelers to come up with a database design that balances the tradeoffs between reporting and loading *Decreased formality in interactions across skill sets LEVEL 3 *All team members can independently complete any task from database design to report creation It’s not about getting your job done it’s about getting the job done
- Frequently and consistently support the team’s self organized culture let the team define their own success criteria don’t overly proscribe HOW things must be done, for example, a source to target mapping may be required, but let the team decide how that information gets captured Determine where there are skill set gaps obtain training to fill the gaps and/or pair up team members to cross train each other Have informal lunch and learns to share knowledge and perspectives, building team communication and trust Create a 5 – 7 person team with a mix of all the skill sets needed to take data from a source system and get it into the hands of end users that need it to make decisions
- Who is on your team? What are their skills, and are they able to perform work cross functionally? For example, can your data architects also write ETL code? Can your ETL programmers also design database tables?
- What aspects are hardest to change? For example, is it so hard to get access to data that when you do, you pull it all in regardless of end user need? Do you have so many layers that it’s hard to imagine updating all of them when you need to make a single change?
- Survey – show of hands
- LEVEL 1 *Numerous (possibly) redundant layers (staging, ODS, EDW, data marts, etc.) *Inclusion of data with no clear business value *Lingering tables, reports, ETL scripts, with no known purpose LEVEL 2 *Active effort to clarify understanding of data’s business value *Streamlined architecture where possible *Process to deprecate unused components LEVEL 3 *Clear understanding of data’s business value *Clear understanding of the purpose for each component of the technical architecture
- How is work getting done now? Who is doing what? How do you complete a full “slice” of functionality needed by end users? If there is a choice between refactoring an existing column and adding a new one that is almost identical – choose refactoring!
- What aspects are hardest to change? For example, is it so hard to get access to data that when you do, you pull it all in regardless of end user need? Do you have so many layers that it’s hard to imagine updating all of them when you need to make a single change?
- *Building blocks of the system (db create scripts, ETL packages, report files, etc.) are not maintained in any central location nor are they under configuration management *Files are manually copied from one environment to another *Some configuration management (SQL scripts to create all db objects are under CM, but not ETL and report information) *Some automation is in place (perhaps to promote new objects or code to another environment or to test ETL) *End-to-end use of optimal engineering practices *Team self-enforces usage through criteria for completing work
- new and changed tables, ETL code, etc. to another environment Create test data for an existing or new ETL process and have scripts that check the load for expected results. Add this for all ETL over time.
- Assuming Agile/Scrum... a typical user story will include or work together to derive a few examples. BI/DW teams may find this particularly challenging because we typically break our work down into technical tasks sequenced logically – database design then ETL then reporting. The focus of each user story should be on providing business value and the team will need to figure out how to achieve that business value in a single sprint. This will involve collaboration between the team and the business to scope the business value in each story to something the team can complete. *The duration of each sprint – two week sprints are most common though up to four weeks is considered acceptable. * The team’s definition of done. Agile software projects focus on delivering working software. The team needs to determine what that means to them on a BI/DW project. At a minimum, the acceptance criteria in a user story must be met, resulting in business value. The team may choose to add additional criteria to their definition of done such as creating standard definitions for any new data added to the DW and updating source-to-target mappings for data transformations.
- What is working well? What isn’t? Are there any bottlenecks occurring? Are there any blockers that keep happening?