Your SlideShare is downloading. ×
Agile Data Management & Integration
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Introducing the official SlideShare app

Stunning, full-screen experience for iPhone and Android

Text the download link to your phone

Standard text messaging rates apply

Agile Data Management & Integration

310
views

Published on


0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
310
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
6
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. By: Marianne Gleason, PMP, CSSBBData Management & Warehouse Consultant
  • 2. DEFINITION OF DATA MANAGEMENTData Management: The business function of planning for, controlling and delivering data and information assets. This function includes: The disciplines of development, execution, and supervision of plans, policies, programs, projects, processes, practices, that control, protect deliver, and enhance the value of data information assets. --- DMBOK DATA MANAGEMENT & INTEGRATION: BUSINESS DILEMMAS 2
  • 3. THE STORY OF TWO LIFECYCLES SYSTEM DEVELOPMENT LIFECYCLE (SDLC) Plan Analyze Design Build Test Deploy Maintain DATA LIFECYCLE Create & Maintain Archive & Plan Specify Enable Purge Acquire & Use RetrieveData is created or acquired, stored and maintained, used, and eventually purged.As I‘m sure many businesses, SMB and Enterprise alike, agree, here’s where it getsinteresting. This is due to the dynamics of data, as it may be extracted, imported, exported,validated, cleansed, transformed, aggregated, analyzed, reported, updated, archived, andbacked up, to name a few, prior to purging. DATA MANAGEMENT & INTEGRATION: BUSINESS DILEMMAS 3
  • 4. HOW DO WE TRANSFORM THE TRADITIONAL LIFE CYCLE TO HANDLE TODAY’S DATA INTEGRATION DEMANDS? WAT E R F A L L METHODOLOGY AGILE METHODOLOGY DATA MANAGEMENT & INTEGRATION: BUSINESS DILEMMAS 4
  • 5. COMPONENTS OF AGILEStory WritingEstimation APPLY TO DATA INTEGRATIONRelease Planning LIFE CYCLE KEYS ARE:Sprint Planning 1. CADENCE 2. CALLABORATIONMetrics 3. COMMUNICATION 4. RISK MITIGATION 5. MINIMIZE DATA TIME TO USE FOR THE BUSINESS DATA MANAGEMENT & INTEGRATION: BUSINESS DILEMMAS 5
  • 6. HOW DOES AGILE APPLY TO DATA INTEGRATION?For the purpose of this presentation, I will be providing examples inrelation to an enterprise data warehouse (EDW). In this case, thedata sets are large, unstructured data which is referring to data thatdoes not fit well into relational database management systems(RDMS). DATA MANAGEMENT & INTEGRATION: BUSINESS DILEMMAS 6
  • 7. EXAMPLE: ADDING COMPLEX DATA FROM A NEW SOURCE INTO THE ENTERPRISE DATA WAREHOUSE (EDW)Below are process steps within an Iteration that integrates with the Agile Components andthe macro Data Integration Life Cycle DATA GOVERNANCE (Meta Data and Document Control) Coding & DataRequirements Data QA & System Transformation Development / Testing / Deployment Profiling Rules & Coding Validation Mappings Rework Rework Rework Rework COMMUNICATION & RISK MITIGATION DATA MANAGEMENT & INTEGRATION: BUSINESS DILEMMAS 7
  • 8. HOW DO WE USE THE AGILE COMPONENTS WITH THE DATA INTEGRATION LIFE CYCLE? • Story Writing C Requirements • Estimation Story Writing O • Estimation M • Release Planning Data Profiling • Spring Planning M Estimation U • Estimation Coding & Data • Release Planning N Transformatio n Rules & • Sprint Planning Mappings Release I Planning C • Release Planning • Sprint Planning Development / A Coding Sprint Planning T • Estimation I QA & System • • Release Planning Sprint Planning Testing / • Metrics O Validation Metrics N • Retrospective / Lessons Learned Deployment • Continuous Improvement DATA MANAGEMENT & INTEGRATION: BUSINESS DILEMMAS 8
  • 9. STORY WRITINGHow does a team determine requirements? Understand the business case / problem statement Draw on team’s expertise to determine tables affected for new data source Data Profiling can assist in determining database tables affected Define all areas of the business affected – Define as Epic vs. Function vs. Task BreakdownTools that can be used: User Stories, Refer to Stakeholder Matrix, Card, User Conversations, Confirmation (Consensus), Acceptance Criteria, System As A Whole Mentality w/in Scope, What/Why/How Personas, Questionnaires, Observations, SMEs, SPIOC Diagrams, Ishikaw Diagrams, RACI Matrix, to name a few DATA MANAGEMENT & INTEGRATION: BUSINESS DILEMMAS 9
  • 10. EPIC STORY WRITING EXAMPLE (SIPOC) =>STORIES FOR LARGE DATA SETS Define the Process Who What is What STEPS are Included WHAT does the WHO are your PROVIDES provided to in the Process today? customer primary the input? START the (high level) receive? (Think of customers? process? their CTQ’s) S p lie up r In u pt P cs ro es O tp t u u C s mr u to e (Who) (Nouns) (Verbs) (Nouns) (Who) Software / Hardware Requirements Cycle Time for Data to Third Party Extract Regulations Use Recipients Vendors Source Input Customer/ Data Profiling Report Generation / Stakeholders (Internal / Data Transportation & Organization External Extracts External) Security Staff Training & Valid / Invalid Data to Government Coding & Data Transformation Rules Regulators Availability (Resources) the Warehouse and Mappings Internal Functions IDS, EDW, Data Mart / Metric Evaluation Development / Coding Vendors affected by data / SMEs Tables Effected Data Analytics Database Environment (Transactional / Mobile Device / Web / Platform(s) QA & System Testing / Validation Customers Analytical) Methodology & Risk Analysis Standards Deployment Process Project / Program Management Testing Results and Plans Evaluations DATA MANAGEMENT & INTEGRATION: BUSINESS DILEMMAS 10
  • 11. ESTIMATION Understand the assumptions and constraints Make sure requirements are understood Understand potential and known areas of rework Use historical throughputs of similar projects Estimations are not contracts – so have cultural flexibility with the team Break down requirement(s) stories into tasks Monitor backlogs throughout iteration => helps for sprint determinationTools That Can Be Used: Poker Planning, Historical Estimates, Velocities for Sprints, Forecasting as a Range/Percentage (Short Term) for sprints and project durations, Project Cost Estimations from Velocity Forecasting, Process Mapping, Hypothesis Statements DATA MANAGEMENT & INTEGRATION: BUSINESS DILEMMAS 11
  • 12. ESTIMATION EXAMPLEThree Components:■ Estimate Size of Stories = Defines Sprint■ Measure Velocity For Each Iteration = Total Sprints Throughput Iteration 1 Forecast:■ Forecast Duration Predict using a Range 5 4 and a % using Project 3 backlog ESTIMATION 2 - Derive Low Velocity (STORY PTS.) 1 - Derive High Velocity 0 - Derive AverageTASK Sprint 1 Sprint 2 Sprint 3 Sprint 4 Define fields to be Velocity mapped (100) - Forecast projectTASK Profile source to duration by # of target data for sprints then convert to mapping / coding complexity SPRINT $/sprint then $/iteration DATA MANAGEMENT & INTEGRATION: BUSINESS DILEMMAS 12
  • 13. RELEASE PLANNING Paradigm shift between traditional plan driven to agility driven from vision and values. Agile Levels: DI Vision, DI Roadmap, Go Live Plan, Iteration Plan, Daily Commitment Set iterations to fit DI Roadmap (usually 1 – 4 week timeframe); decrease data to business use cycle times Connects strategic vision to delivery approach (source to target), Eliminates Waste (rework) / Lean, Eliminates Variation, Better Decision Making, Improves Communication, Improves Morale Release Planning/DI Planning leads to Roadmap, Plan, Backlog Key Elements: Schedule, Estimates on Epics / Stories, Prioritized Backlogs, Velocity of TeamBottom Line to Tools: Complexity is Estimated, Velocity is Measured, Duration is Derived DATA MANAGEMENT & INTEGRATION: BUSINESS DILEMMAS 13
  • 14. RELEASE PLANNING PICTORIALRELEASE / DATA INTEGRATION PHASE 1Iteration 1 Iteration 2 Iteration 3RELEASE / DATA INTEGRATION PHASE 2Iteration Iteration Iteration Iteration Iteration 4 5 6 7 8 DATA MANAGEMENT & INTEGRATION: BUSINESS DILEMMAS 14
  • 15. SPRINT PLANNING● Determine and agree on the sprint and next sprint goals● Determine required attendees, inputs and outputs● Prioritized logs/backlogs and validate based on estimates● Review and seek clarification of stories & tasks● Define and estimate the work plan by breaking into tasks from user stories● Daily Standups● Sprint Review and Demo Integration● Retrospective / Lessons Learned DATA MANAGEMENT & INTEGRATION: BUSINESS DILEMMAS 15
  • 16. EXPANDING ON SPRINT PLANNING ELEMENTS● Participation● Prioritized Backlog● Presentation of Candidates Stories● Agreeing On Sprint Goal● Validation of Sprint Backlog Based on Team Estimation of Stories● Capacity Planning● Defining and Estimating the Work Plan● Daily Stand Up Meetings● Sprint Review and Closeout● Retrospective / Lessons Learned DATA MANAGEMENT & INTEGRATION: BUSINESS DILEMMAS 16
  • 17. METRICS● Derive measurements (Quantitative/Qualitative)● Leading / Lagging measurements● Metrics must be motivational and informative● Determine whether tasks are done – either 100% complete or not complete● Some agile metrics (going beyond common metrics): ■ Velocity – Sum of points delivered for each iteration / # of iterations ■ Burndown – Rate at which requirements are being delivered ■ Burnup – Project story points are being met – (i.e. scope) ■ Cumulative Flowcharts – The requirements are in respect to the lifecycle over time (i.e. Not Started, In Progress, Pending Acceptance, Completed) Leads to more accurate OLAP and/or OLTP for BI and Analytic results in conjunction with the company’s business model and dynamic efforts regarding data management strategic planning efforts. DATA MANAGEMENT & INTEGRATION: BUSINESS DILEMMAS 17
  • 18. EXAMPLES OF AGILE METRICS - BURNDOWN 90 80 70 60% COMPLETE 50 Ideal 40 30 Actual 20 10 0 1 2 3 4 5 Iterations DATA MANAGEMENT & INTEGRATION: BUSINESS DILEMMAS 18
  • 19. QATesting Defects Pareto Chart 120% 100% 80%Frequency % 60% Cumulativ e % 40% 20% 0% Mapping Coding Target Meta Data Data Joins Data Type Foreigh Grouping Wrong SK Domains Standards Key Value Unclear Cause Lookup DATA MANAGEMENT & INTEGRATION: BUSINESS DILEMMAS 19
  • 20. EXAMPLES OF AGILE METRICS - BURNUP DATA MANAGEMENT & INTEGRATION: BUSINESS DILEMMAS 20
  • 21. EXAMPLES OF AGILE METRICS - ITERATIONIf backlog is sized at 60 storyPoints, using this velocity trend COST USING VELOCITYThe projected duration is:Range: Iteration - Duration EstimateLow Velocity: 10 story pointsHigh Velocity : 30 story pointsAverage Velocity: 20.5 story points 30 25The team’s velocity ranged from10 to 30 story points. 20 15 Estimate60/10 = 6 sprints 1060/30 = 2 sprints 5Backlog will release between 2 and 06 sprints Sprint 1 Sprint 2 Sprint 3 Sprint 4Notice Sprints 1 and 2 have a high degree of story point variability, as If cost per sprint is $10,000 then iteration range prediction is:the team is likely in the Forming/Storming team development stages.Sprints 3 & 4 tend to be closer in story points, as the team begins to Low Estimate: (2 sprints)(10,000) = $20,000 High Estimate: (6 sprints)(10,000) = $60,000attain the Norming/Performing team development status. Avg. Estimate (2.9 sprints)(10,000) = $29,000 DATA MANAGEMENT & INTEGRATION: BUSINESS DILEMMAS 21