Agile BI/DW Daniel Fagerström [email_address]

Experience Used agile methods 10 years Developed BI/DW 5 years Led large BI/DW project during last year using Scrum Database lead at alaTest.com

Used agile methods 10 years

Developed BI/DW 5 years

Led large BI/DW project during last year using Scrum

Database lead at alaTest.com

Why agile? Deliver early Reduce risk Make the project steerable and transparent Make development efficient and fun

Deliver early

Reduce risk

Make the project steerable and transparent

Make development efficient and fun

How?

Just follow the book? Yes, but … What are the user stories? Agile DB practices? Agile operations?

Yes, but …

What are the user stories?

Agile DB practices?

Agile operations?

Overview Before starting Project cycles Agile planning Agile DB practices Agile operations

Before starting

Project cycles

Agile planning

Agile DB practices

Agile operations

Before the project Introduce agile methods while selling Agile architecture, dimension models Agile estimation based offering

Introduce agile methods while selling

Agile architecture, dimension models

Agile estimation based offering

Sprint 0 Collocation (set up team room) Appropriate HW, SW, test server Involve operations Contact the data source owners

Collocation (set up team room)

Appropriate HW, SW, test server

Involve operations

Contact the data source owners

Cycles Release cycle Envision & fund Plan Build Deploy & review Sprint Plan Develop Demo Review Daily Scrum Sprint Daily Scrum Sprint Planning Sprint Demo Sprint Retrospective Release cycle

Release cycle

Envision & fund

Plan

Build

Deploy & review

Sprint

Plan

Develop

Demo

Review

Daily Scrum

Planning DW for loyalty card company Epic As marketing manager I want to use the DW to understand customer behavior Theme As a campaign manager I want to follow sale over time for a certain store User story I want the customer dimension to be updated daily

DW for loyalty card company

Epic

As marketing manager I want to use the DW to understand customer behavior

Theme

As a campaign manager I want to follow sale over time for a certain store

User story

I want the customer dimension to be updated daily

Planning (decomposition) Architecture driven story decomposition See Agile Data Warehousing, Ralph Huges Use course architecture as common language for Product owner and developers Architectural dimensions Subject area, Target layer, Transformation, type, Refresh type, Refresh frequency, Automation, User friendliness Each user story must enable feedback

Architecture driven story decomposition

See Agile Data Warehousing, Ralph Huges

Use course architecture as common language for Product owner and developers

Architectural dimensions

Subject area, Target layer, Transformation, type, Refresh type, Refresh frequency, Automation, User friendliness

Each user story must enable feedback

Dimension model Customer name city Store name region Product name class Date year month ReceiptLine amount

Planning (decomposition) Start with the most important Star (subject area) Dimension Attributes in the dimension

Start with the most important

Star (subject area)

Dimension

Attributes in the dimension

Architecture Customer registry Product registry Org. registry Transactions Raw data Staging Dimension model Marts OLAP Cube Reporting Applications ETL

Planning (decomposition) ETL – Start with simpler/earlier tasks Transformation type Direct data -> aggregations -> business rules Refresh frequency One time load -> monthly -> weekly -> daily Target layer Raw -> staging -> presentation -> end user view Refresh type Link -> snapshot -> incremental

ETL – Start with simpler/earlier tasks

Transformation type

Direct data -> aggregations -> business rules

Refresh frequency

One time load -> monthly -> weekly -> daily

Target layer

Raw -> staging -> presentation -> end user view

Refresh type

Link -> snapshot -> incremental

Planning (decomposition) BI Transformation type Refresh frequency Automation On demand on work station -> on demand posted on server -> scheduled User friendliness Single table -> linked table -> navigation -> pick list -> dash board Using expert analysis tools -> reports

BI

Transformation type

Refresh frequency

Automation

On demand on work station -> on demand posted on server -> scheduled

User friendliness

Single table -> linked table -> navigation -> pick list -> dash board

Using expert analysis tools -> reports

Breadth or depth? Breadth Simplified load of the most important dimension models Early feedback, earlier build of dependent systems Depth Complete load of one dimension at a time Early deployment of usable sub systems

Breadth

Simplified load of the most important dimension models

Early feedback, earlier build of dependent systems

Depth

Complete load of one dimension at a time

Early deployment of usable sub systems

Agile DB Practices Agile DB refactoring Scott Ambler Change scripts for everything Common code ownership Everybody can can change the models DB models in visual modeling tool Don't become a bottleneck Broad contact area between team members and users, operations and source data specialists

Agile DB refactoring

Scott Ambler

Change scripts for everything

Common code ownership

Everybody can can change the models

DB models in visual modeling tool

Don't become a bottleneck

Broad contact area between team members and users, operations and source data specialists

Agile DB Practices Course architecture Agile modeling Just in time design Agile documentation

Course architecture

Agile modeling

Just in time design

Agile documentation

Agile DB Practices DB Schema in repo Subset data Your own DB and development sandbox Staged deployment: dev -> test -> prod Continuous deployment TDD?

DB Schema in repo

Subset data

Your own DB and development sandbox

Staged deployment: dev -> test -> prod

Continuous deployment

TDD?

Agile Operations Involve operations from day one Operations part of the project Deploy at demo/production server often Start source data deliveries ASAP

Involve operations from day one

Operations part of the project

Deploy at demo/production server often

Start source data deliveries ASAP

Agile Operations -> Operations, support and maintenance during the project Kanban for that? Project time #Features in prod.

Kanban for that?

Conclusions Some mental barriers for DB/DW/BI-developers Transparent Steerable Works great! Happy customers (and developers)!

Some mental barriers for DB/DW/BI-developers

Transparent

Steerable

Works great!

Happy customers (and developers)!

Literature

www.danielfagerstrom.com