Agile Data Warehousing (DW)/Business Intelligence (BI): Addressing the Hard Problems

Agile DW/BI
Addressing the Hard Problems
Scott W. Ambler
Consulting Methodologist
Ambysoft Inc.
© Ambysoft Inc. AgileData.org AgileModeling.com 1

Scott Ambler
• scott@scottambler.com
• Twitter: @scottwambler
• linkedin.com/in/sambler/
Ambysoft.com
Thought Leader
AgileData.org
Thought Leader
AgileModeling.com

Agenda
• Your assignment
• Agile Data ways of thinking (WoT)
• What is agile DW/BI?
• What are the “hard problems”?
• How to address each problem
• Parting thoughts
• Questions and answers

Identify the questions you want
answers for over the next two
days
Your Assignment

Agile Data Ways of Thinking (WoT)

Agile Data Ways of Thinking (WoT)
AgileData.org/essays/philosophies.html
Look Beyond Data
Collaborate Closely
Be Quality Infected
Embrace Evolution
Be Enterprise Aware
Fit-For-Purpose
Agile Enterprise Groups

Agile Data Warehousing (DW)
and Business Intelligence (BI)

Agile DW/BI is the act
of providing quality
information in a
collaborative and
evolutionary manner.

The
“Hard Problems”
in Agile DW/BI

The “Hard Problems”
1. Up-front data architecture work takes too long
2. We need to deliver value every sprint
3. We need to install required infrastructure
4. We don’t have enough agile data people
5. Sprints are too slow, we need to go faster
6. Data analytics often takes longer than a single sprint
7. My product owner doesn’t understand the data
8. We have too much data technical debt
9. Our stakeholders don’t want to work with us

Problem: Up-Front Architecture

DV2 Has Done a Lot of Thinking for You
Used with permission from Dan Linstedt and DataVaultAlliance Holdings, LLC - DataVaultAlliance.com

Key Model: Data Source Diagram
• Identify main data sources
• Goal is to understand the
physical landscape
• Support with key source
information:
• Owner
• Access strategy
• Main entity types
• Quality rating
• Refresh rate
NCC
1701
Cust
THX
1138
Lead
File
…
JB7
File
Social M
edia Data
…
Data Vault

Key Model: Conceptual Diagram
• Identify main business
terms (entity types) and
their relationships
• Goal is to understand the
conceptual landscape
• Support with source to
entity type mappings,
tying business terms to
business processes

Just Barely
Good Enough
(JBGE) Artifacts
Source: AgileModeling.com/essays/barelyGoodEnough.html
Risk
Complexity
Desire for “predictability”
More modeling Less modeling
Implementer experience & skill
Access to stakeholders
Ease of change
Uncertainty
Collaborative WoW

Problem: Deliver Value
Every Sprint

Each sprint a disciplined agile team produces a working solution
Functionality is added in “vertical slices”
Example slicing strategies for DW/BI:
• One new data element from a single data source
• One new data element from several sources
• A change to an existing report
• A new report
• A new reporting view
• A new data mart table
AgileData.org/essays/verticalSlicing.html
Vertical Slicing

Source: AgileData.org/essays/questionStories.html
Vertical Slices for DW/BI: Question Stories
A question story is a specialized user story specific to data-oriented requirements.
Examples:
• As a sales manager I want to know the level of sales by my team by the end of each day so that I know where we
stand.
• As an instructor I want to know the certification pass rate of my students so that I can update my seminar marketing
message.
• As a restaurant owner I want to know the common combinations of menu options being ordered so that I can
identify potential specials.
• As a city councillor I want to know the number of complaints about road quality so that I can determine where we
need to focus our repair efforts.
Formats:
As a [Role] I want to know [Something] by [Timeframe] because [Reason]
- OR -
As a [Role] I want to know [Something] because [Reason]

Problem: Installing the Required
Infrastructure

Adopt the Great DV2
Vendor Solutions

Embrace Evolution
• Recognize that:
• You don’t need everything right
away
• Your situation will change
• Evolutionary development
• Think about the future, but wait
to act

Sometimes The First Release
Is Bigger Than You’d Normally Prefer

Problem: Lack of
Agile Data People

Agile Data Roles
Source: AgileData.org/essays/roles.html

Generalizing Specialists
Generalist
Specialist
Generalizing
Specialist
Broad
knowledge
Deep
skills
Source: http://agilemodeling.com/essays/generalizingSpecialists.htm
Skill
1
Skill
2
Skill
4
Skill
N
Skill
3
Skill
5
Skill
6
…
Knowledgeable
Competent
Specialist
Also known as:
• T-skilled people
• Cross-functional people

Invest in Your People
• Training
• Coaching
• Collaborative, non-solo
work

Problem: Sprints Are Too Slow

Run the First Release as an Agile Project
Source: http://agiledata.org/essays/disciplinedAgileDW.html

Evolve via Lean Continuous Delivery
Source: http://agiledata.org/essays/disciplinedAgileDW.html

Also….
Negotiate stakeholder expectations
• Perhaps the real issue is
misalignment
Shorten your sprints
• Motivates you to squeeze waste out
of your way of working (WoW)
• Gets you closer to a continuous
delivery strategy

Problem: Data Analytics
Takes Longer Than a Sprint

Look-Ahead Data Analysis: Agile
Scenario: You want to implement three question stories in sprint #9
You need to:
• Have a definition of ready (DoR) indicating the amount of data analysis work required
• Guesstimate the amount of data analysis required for each one, and then perform the
analysis sufficiently before sprint #9
• Have sufficient capacity to perform look-ahead data analysis
• Interleave data analysis for other sprints into the work of the people performing it
Note: Staffing your team with specialists will exacerbates work scheduling challenges.
Source: AgileData.org/essays/lookAheadDataAnalysis.html

Look-Ahead Data Analysis: Continuous Delivery
Development – QS 9a
Development – QS 9b
Development – QS 9c
Scenario: You want to implement the same three question stories
• You are not constrained by organizing the work into sprints
• Development = Data analysis + other implementation work
• Work can be brought into the team as capacity permits
• Value is delivered when it is available
• Average cycle time to deliver stories is shorter
• Staffing your team with generalizing specialists may enable you to swarm around data
analysis work and shorten the development time of a question story
Source: AgileData.org/essays/lookAheadDataAnalysis.html

Problem: My Product Owner
Doesn’t Understand the Data

Product Owners Must Collaborate Closely
With Agile Data Professionals

Active Stakeholder Participation
Source: AgileModeling.com/essays/activeStakeholderParticipation.htm
• Adopt inclusive modeling tools and
techniques and enable stakeholders to
explore their own requirements
• Product Owners, like Business Analysts,
can miss or filter out important details
• Supports a strategy of self-serve data
• Requires stakeholders to make time

Also…
Train/coach the product owner
• Help them get the knowledge that they
need
Generalizing specialists
• It’s not just an issue with product owners,
everyone should have a better data
background

Problem: We Have Too Much
Data Technical Debt

Be Quality
Infected!
Source: AgileData.org/essays/dataTechnicalDebt.html
Data technical debt refers to quality
challenges associated with data sources.
Data technical debt impedes the ability of
your organization to leverage information
effectively for informed decision making,
increases operational costs, and impedes
your ability to react to changes in your
environment.

Source: AgileData.org/essays/databaseRefactoring.html
Customer
CustomerID <<PK>>
Fname
Customer
CustomerID <<PK>>
Fname
FirstName
SynchronizeFirstName()
Customer
CustomerID <<PK>>
FirstName
Original
Schema:
Interim
Schema:
Final
Schema:
Database Refactoring
A database refactoring is a simple change to a
database schema that improves its design while
retaining both its behavioral and informational
semantics
A database schema includes structural aspects such
as table and view definitions; functional aspects such
as stored procedures and triggers; and informational
aspects such as the data itself

Problem: Our Stakeholders
Don’t Want to Work With Us

Deliver Value Frequently
Frequent Value Delivery
• Value delivered from very early on
• Provides opportunities to gain
concrete feedback and to steer
• Flexible, enabling the team to focus
on regularly delivering most
valuable functionality
• Requires team to be skilled and
disciplined
€
“Big Release” Value Delivery
• Convenient for the IT team
• High-risk, likely leading to something
that doesn’t meet their actual needs
• Very likely to go late, be over budget,
and potentially cancelled before
delivery
€ € € € € € € € € € € € €

Also…
Question stories
• Focus on their needs, not yours
Active stakeholder participation
• Make it more enjoyable for them

Parting Thoughts

The increasing pace of change,
increasing complexity,
and increasing volume of data
demands nothing less than
complete data agility

What questions do you
want answers for over the
next two days?

Thank You!
• scott@scottambler.com
• Twitter: @scottwambler
• linkedin.com/in/sambler/
Ambysoft.com
Thought Leader
AgileData.org
Thought Leader
AgileModeling.com

Backup Slides

Critical Agile Data WoW
• Active stakeholder participation
• AgileModeling.com/essays/activeStakeholderParticipation.htm
• Look-ahead data analysis
• AgileData.org/essays/lookAheadDataAnalysis.html
• Question stories
• AgileData.org/essays/questionStories.html

Agile Enterprise Architecture
AgileData.org/essays/enterpriseArchitecture.html
• Collaborative
• Enterprise focused
• Architects also implement
• Evolutionary
• Consumable

The Agile Database Techniques Stack - AgileData.org/essays/techniquesStack.html
Vertical Slicing
Clean Architecture and Design
Agile Data Modeling
Database Refactoring
Database Regression Testing
Continuous Database Integration and
Deployment
Configuration Management
Organize value into small increments
Enabled by
Evolved via
Improves via
Verified via
Automated via
Pulls files from
High quality, evolvable, and flexible - Agile
Iterative, incremental, and collaborative
Safely improves quality of data sources
Verifies and enables safe evolution
Fundamental DevOps infrastructure
Manage data assets as assets

Agile Data Warehousing (DW)/Business Intelligence (BI): Addressing the Hard Problems

More Related Content

Similar to Agile Data Warehousing (DW)/Business Intelligence (BI): Addressing the Hard Problems

More from Scott W. Ambler

Recently uploaded

Agile Data Warehousing (DW)/Business Intelligence (BI): Addressing the Hard Problems