1. Agile Data Governance
Tami Flowers
Director, Governance Solutions
MetaGovernance Solutions, LLC
DGIQ June 2016
2. Who am I?
Tami Flowers
Director, Governance Solutions at MetaGovernance
PMI-PMP, PMI-ACP, and Certified Scrum Master
Agile Coach
Developer Project Manager
Enterprise
Architect
Data Governance
Consultant
MetaGovernance Solutions LLC. All Rights Reserved. 2016
3. Today
Data Governance
Agile
How to meld them together
A few minutes on the topic
An “experience”
Debrief/Reflection
MetaGovernance Solutions LLC. All Rights Reserved. 2016
4. Experience: Ice Breaker
1. Write your answer to the following questions on sticky
notes. One answer per note. As many answers as you wish.
What are you hoping to learn today?
What specific questions are you hoping get answered today?
2. Discuss amongst your team. Group like answers together.
3. Have each person mark their level of experience with the
following:
MetaGovernance Solutions LLC. All Rights Reserved. 2016
No experience Some
experience
Quite a bit of
experience
Agile
Data Governance
5. Data Governance
Two viewpoints
Establishing a Data Governance Organization
Framework
Implementing Data Governance actionable Data
Governance
MetaGovernance Solutions LLC. All Rights Reserved. 2016
6. Data Management Functions
Data
Governance
Data
Architecture
Data
Development
Data
Operations
Data Security
Reference
and Master
Data
EDW and BI
Document &
Content
Metadata
Data Quality
Data Governance is at the heart of all Data Management activities.
As defined by the Data Management Association – Data Management Body of Knowledge (DAMA-DMBOK)
MetaGovernance Solutions LLC. All Rights Reserved. 2016
8. Data Governance Organization Framework
Executives
Business & Technical
Members
Stewards/analysts/architects/
administrators/engineers
Consumers
Data Governance Council (DGC)
Data Governance
Steering Committee
(DGSC)
Data Stewardship
Teams (virtual)
• Establish & train DGC members
• Agree on Data Governance roadmap
• Determine members & empower DGSC
• Define Data Governance policy
• Meet on recurring basis
• Receive training and education
• Execute DG roadmap
• Define DG communication plan
• Select & empower stewardship teams
• Define roles and responsibilities
• Define charter
• Meet on recurring basis
• Technical and Business Members
• Receive training and education
• Execute DG roadmap
• Business Members
• Receive training and
education
End Users
MetaGovernance Solutions LLC. All Rights Reserved. 2016
9. Data Governance Organization Framework
Actionable Data Governance
Executives
Business & Technical
Members
Stewards/analysts/architects/
administrators/engineers
Consumers
Data Governance Council (DGC)
Data Governance
Steering Committee
(DGSC)
Data Stewardship
Teams (virtual)
• Lead and promote program
• Empower DGSC
• Arbitrate data related issues, as needed
• Execute DG communication plan
• Define data management policies, procedures and
standards
• Define stakeholders, and sources and use of data
per subject area
• Approve and oversee corrective actions
• Technical and Business Members
• Implement policies, procedures,
standards and controls
• Identify and report data issues
• Corrective action input & execution
• Corrective action status reporting
• Business Members that
consume and use data
• Identify and report data
issues
End
Users
MetaGovernance Solutions LLC. All Rights Reserved. 2016
10. Registered Data Governance Stakeholders
Data Subject
Area ID
Major Subject
Group 1
st
Sub Grouping 2
nd
Sub Grouping
Accounting
Operations
Collateral
CreditRisk
Finance
FinancialReporting
InternalAudit
MarketRisk
Sales
LoanOperations
Compliance
Information
Technology
1 Trade Loan Adjustable-Rate
Loan
2 Trade Loan Fixed-Rate Loan
3 Trade Loan Structured Loan
4 Trade Loan Amortizing Loan
5 Trade Loan Letters of Credit
6 Trade Loan Prepayment Fee
11 Financial Accounting Accrued Interest
12 Financial Accounting Outstanding
Principle
LEGEND
CUSTODIAN
CONSUMER
STEWARD
DELEGATE
MetaGovernance Solutions LLC. All Rights Reserved. 2016
12. The Agile Manifesto
We are uncovering better ways of developing software by doing it and helping
others do it. Through this work we have come to value:
Individuals and interactions over Processes and tools
Working software over Comprehensive documentation
Customer collaboration over Contract negotiation
Responding to change over Following a plan
That is, while there is value in the items on the right, we value the items on the
left more.
In February 2001, 17 software developers met at the Snowbird resort in Utah to discuss lightweight development methods.
They published the Manifesto for Agile Software Development.
MetaGovernance Solutions LLC. All Rights Reserved. 2016
13. Agile Principles
1. Our highest priority is to satisfy the customer
through early and continuous delivery
of valuable software.
2. Welcome changing requirements, even late in
development. Agile processes harness change for
the customer's competitive advantage.
3. Deliver working software frequently, from a
couple of weeks to a couple of months, with a
preference to the shorter timescale.
4. Business people and developers must work
together daily throughout the project.
5. Build projects around motivated individuals.
Give them the environment and support they need,
and trust them to get the job done.
MetaGovernance Solutions LLC. All Rights Reserved. 2016
14. Agile Principles
6. The most efficient and effective method of
conveying information to and within a development
team is face-to-face conversation.
7. Working software is the primary measure of progress.
8. Agile processes promote sustainable development.
The sponsors, developers, and users should be able
to maintain a constant pace indefinitely.
9. Continuous attention to technical excellence
and good design enhances agility.
10. Simplicity--the art of maximizing the amount
of work not done--is essential.
11. The best architectures, requirements, and designs
emerge from self-organizing teams.
12. At regular intervals, the team reflects on how
to become more effective, then tunes and adjusts
its behavior accordingly.
MetaGovernance Solutions LLC. All Rights Reserved. 2016
15. Experience: Agile Principles
How can we apply these principles to establishing a data
governance framework
How can we apply these principles to actionable data
governance (day-to-day governance)?
MetaGovernance Solutions LLC. All Rights Reserved. 2016
19. Experience: Agile Practices to
Lean Principles
Agile Practice/Lean
Principle
Eliminate
Waste
Empower
theTeam
Deliver
Fast
Optimize
theWhole
Build
Qualityin
Defer
Decisions
Amplify
Learning
Teams make own
decisions
Just in time planning
Team retrospectives
Two week iterations
Unit test as we go
Shadow the business to
learn what they do
MetaGovernance Solutions LLC. All Rights Reserved. 2016
20. Experience: Data Governance to
Lean Principles
Agile Practice/Lean
Principle
Eliminate
Waste
Empower
theTeam
Deliver
Fast
Optimize
theWhole
Build
Qualityin
Defer
Decisions
Amplify
Learning
Teams make decisions
Just in time planning
Retrospectives
Frequent iterations
Validate/approve as we go
Involve the business
stakeholders
MetaGovernance Solutions LLC. All Rights Reserved. 2016
21. Kanban
Capture the work
Make it visible
Each story is about the same size
Limit the work in progress (WIP)
MetaGovernance Solutions LLC. All Rights Reserved. 2016
22. Experience: WIP
8-10 people per team
4 workers – will do the work
4 managers – will watch over their worker
1-2 timers – will time how long each round takes
This is called the Penny Game and is available on
www.tastycupcakes.org
MetaGovernance Solutions LLC. All Rights Reserved. 2016
23. Commonalities of all
Agile/Lean flavors
Capture the work
Make it visible
Minimally Viable Product
(MVP)
Engage the owner/customer/
stakeholders
Have owner prioritize the
work
Work on the highest priority
work first
Iterate
Continually retrospect and
improve
MetaGovernance Solutions LLC. All Rights Reserved. 2016
24. Roles
Stakeholders
Product owner is one
All have a stake in the project
DGSC members
Product Owner
Chairperson of DGSC
Has the vision and can convey it to the team and stakeholders
Prioritizes backlog
Provides direction to and is engaged with the team
Scrum Master – facilitator
Bulldozer: Remove roadblocks
Servant leader: Do anything needed to help team perform at their highest level
Stewardship Team Roles
Stewards
Developers/Engineers/Architects/Administrators
Business Analysts/Quality Assurance Analysts
MetaGovernance Solutions LLC. All Rights Reserved. 2016
25. Ceremonies
Ceremonies Traditional
Agile/Lean Project
Data
Governance
Framework
Actionable Data
Governance
Planning Meetings Start of every
sprint/iteration
DGSC or
subgroup
meetings
Start of every
sprint/iteration
Scrums/Stand Ups Daily, 15 minutes 1-3 times a
week
Daily, 15 minutes
Review meetings End of every
scrum/iteration
DGSC or
subgroup
meetings
End of every
sprint/iteration
Retrospectives End of every
sprint/iteration
End of every
sprint/iteration
End of every
sprint/ iteration
Backlog Grooming In the week
between
sprints/iterations
In between
sprints/
iterations
In the week
between
sprints/iterations
MetaGovernance Solutions LLC. All Rights Reserved. 2016
26. 4 quadrants to help with
backlog grooming
Broccoli Organic Beef
Twinkies Éclair
Effort
Value
One of our classmates shared this quadrant with us during a discussion. They use this
in their organization to help discuss the value of a story in the backlog and it helps
them select stories for the sprint/iteration.
MetaGovernance Solutions LLC. All Rights Reserved. 2016
27. Artifacts
Artifacts Traditional
Agile/Lean
Data Governance
Framework
Actionable Data
Governance
Product Backlog Created from
conversations
with stakeholders
Created from
Data Governance
Roadmap
Created from DG
policies and
procedures
Sprint/Iteration
Backlog
Prioritized by
product owner
during planning
meeting
Prioritized by
DGSC
chairperson
during planning
meeting
Prioritized by
product owner
during planning
meeting
Stories Created from
conversations
with stakeholders
Created from
Data Governance
Roadmap
Created from DG
policies and
procedures
MetaGovernance Solutions LLC. All Rights Reserved. 2016
28. Data Governance artifacts
within a project
Stories
As a <role> I want <what> so that <business value>
If not a story, include data governance as acceptance
criteria in stories
As an accountant, I want current balances for loans, so that I
can complete daily reporting
All data elements are defined in the business dictionary
Data profiling of the balances occurs when loading into the EDW
An automated control is added so the current balance is
automatically reconciled between the G/L and Loan System
MetaGovernance Solutions LLC. All Rights Reserved. 2016
29. Reality Check
Agile isn’t a silver bullet that will magically fix
everything
New, unexpected challenges will occur
Teamwork
Soft skills
Growing pains
MetaGovernance Solutions LLC. All Rights Reserved. 2016
30. Reality Check
Data Governance isn’t a silver bullet that will be easy to
quickly implement
Must include the business
Must be added to existing processes to be successful
Lots of training and education is needed
Executive support is critical
Will uncover data issues that some don’t want uncovered
MetaGovernance Solutions LLC. All Rights Reserved. 2016
31. Experience: Adding Governance
to a Project
Identify what is needed to add governance to a project.
What 2-3 items are the highest priority?
Project: bring new data source in
Needs: Ensure all registered stakeholders and their data
governance roles are known. Identify and define business
attributes. Profile data for potential issues.
MetaGovernance Solutions LLC. All Rights Reserved. 2016
32. Group experience: What stories are
needed if we are bringing in a new
data source?
Identify data quality dimensions and thresholds
Identify archiving and data retention needs
Classify the data for visibility and security
Identify data stewards
Establish profiling and cleansing rules
Establish SLAs with the provider
Identify legal aspects, regulatory impacts
Identify restrictions on the data, permissible use
Evaluate risk involved with data
Determine requirements to match to master or reference data; identity attributes
Determine upwards and downward impacts to business processes
Configuration management; hardware/software needed, documentation
Define metadata, definitions, associations.
MetaGovernance Solutions LLC. All Rights Reserved. 2016
33. Registered Data Governance Stakeholders (we
also want to identify and record this metadata
each iteration/sprint)
Data Subject
Area ID
Major Subject
Group 1
st
Sub Grouping 2
nd
Sub Grouping
Accounting
Operations
Collateral
CreditRisk
Finance
FinancialReporting
InternalAudit
MarketRisk
Sales
LoanOperations
Compliance
Information
Technology
1 Trade Loan Adjustable-Rate
Loan
2 Trade Loan Fixed-Rate Loan
3 Trade Loan Structured Loan
4 Trade Loan Amortizing Loan
5 Trade Loan Letters of Credit
6 Trade Loan Prepayment Fee
11 Financial Accounting Accrued Interest
12 Financial Accounting Outstanding
Principle
LEGEND
CUSTODIAN
CONSUMER
STEWARD
DELEGATE
MetaGovernance Solutions LLC. All Rights Reserved. 2016
34. Data Governance (Overall Framework)
Provide data governance common
vocabulary
Define key governance roles
Define roles & responsibilities of key gov
roles
Define data governance policy
Define concept of data governance
controls
Project Specific Data Governance –Risk Project
Confirm data subject areas within project
scope
Confirm system of record for each subject
area
Define data quality issues and standards
Define data quality testing & remediation
strategy
Confirm data gov roles for each subject area
Determine existing data controls for each sub
area
Determine recommended data gov controls
Define recommended data gov control &
quality implementation priority
Obtain DGWG approval
FEB MAR APR MAY JUN JULJAN AUG SEP
Risk Project Timeline (sample timeline)
MetaGovernance Solutions LLC. All Rights Reserved. 2016
35. Experience: Setting up a
Governance Framework
Identify what is needed to setup a governance framework.
Generally this is done by looking at the data governance
roadmap (see next slide for example)
Have the product owner prioritize the first 3 items.
MetaGovernance Solutions LLC. All Rights Reserved. 2016
36. Data Governance (DG) Framework
Establish Data Governance Council (DGC)
- Identify and appoint members
- Train and educate DGC
- Agree on common data governance vocabulary
- Agree on common DG stakeholder roles
- Agree on roles and responsibilities for stakeholders
- Agree on scope, authority and structure of DGC
- Agree on initial DGSC members
- Define DGC charter and guiding principles
- Agree on key metrics to evaluate governance
- Participate in recurring DGC meetings
- Champion and promote DG
Establish Data Governance Steering Committee
- Identify and appoint members
- Train and educate DGSC
- Define common data governance vocabulary
- Define key DG stakeholder roles
- Define roles and responsibilities for DG roles
- Define scope, authority and structure of DGSC
- Identify and appoint data stewards
- Agree on DGSC charter and guiding principles
- Define objectives of DGSC
- Identify Governance Coordinator for DG activities
- Participate in recurring DGSC meetings
- Champion, communicate and promote DG
Data Governance Framework Sample Timeline
Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8
MetaGovernance Solutions LLC. All Rights Reserved. 2016
37. Ways Agile can help
Increased communication and visibility
Increased teamwork
More frequent delivery of business value
Implement governance on high priority items first, show
success, move on…continuous improvement
MetaGovernance Solutions LLC. All Rights Reserved. 2016
38. Continuous improvement
Use retrospectives to make your process better
It’s ok to add just 1 or 2 new things at a time
Get help if you need it, lots of resources and experts are out
there
Mentoring and coaching will be critical
MetaGovernance Solutions LLC. All Rights Reserved. 2016
46. Retrospectives
At end of each sprint/iteration
What went well – keep doing
What didn’t go well – stop doing
What could go better – change it
Entire team attends
MetaGovernance Solutions LLC. All Rights Reserved. 2016
47. Experience: Retrospective
What went well
What didn’t go well
What could go better
MetaGovernance Solutions LLC. All Rights Reserved. 2016
48. Lessons learned
There isn’t one definition of Agile; all companies tend to do it
differently. That’s ok. Make it what works best for your
organization.
If you are setting up governance for the first time, get the
framework in place first. Then determine low hanging fruit, such
as data quality issues, and implement governance starting
there.
Data Governance MUST involve the business; it can’t just be an
IT focus.
A lot of training and education is needed for governance.
Data Governance must be added to existing processes and
procedures in order to give it teeth.
SDLC
Architecture reviews
MetaGovernance Solutions LLC. All Rights Reserved. 2016
49. Good resources for Agile and
Team Building
Mike Cohn – MountainGoatSoftware.com
PMI-ACP Exam Prep – Mike Griffiths
Johanna Rothman books: http://www.jrothman.com/books/
The Human Side of Agile, Gil Broza
Agile Retrospectives: Making Good Teams Great, Esther Derby and Diana Larsen
Crucial Conversations, Kerry Patterson
Crucial Confrontations, Kerry Patterson
Influencer
The Deadline: A Novel about Project Management, Tom DeMarco
Teamwork is an Individual Skill, Christopher Avery
Behind Closed Doors: Secrets of Great Management, Johanna Rothman and Esther
Derby
DGIQ presentations: Roberta, Maggie from GM
MetaGovernance Solutions LLC. All Rights Reserved. 2016
50. Me
Twitter: TamiLFlowers
LinkedIn: Tami Flowers
Email: tflowers@metagovernance.com
Slideshare: www.slideshare.nettamiflowers
www.metagovernance.com (blog)
Blog for DataVersity
Thanks!
MetaGovernance Solutions LLC. All Rights Reserved. 2016
Editor's Notes
The meanings of the manifesto items on the left within the agile software development context are:
Individuals and interactions: in agile development, self-organization and motivation are important, as are interactions like co-location and pair programming.
Working software: working software will be more useful and welcome than just presenting documents to clients in meetings.
Customer collaboration: requirements cannot be fully collected at the beginning of the software development cycle, therefore continuous customer or stakeholder involvement is very important.
Responding to change: agile development is focused on quick responses to change and continuous development.[6]