Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Lean Modeling for Any Methodology

406 views

Published on

With changes in software development methodologies, the role of the data modeler has changed significantly. In many organizations, data modelers now find themselves on the outside looking in, relegated to documentation “after the fact” rather than active participation in database design where the true value is added. Some organizations using Agile practices have incorrectly dismissed the importance of data modeling, often with disastrous results.
IDERA’s Ron Huizenga will discuss how to adopt a lean data modeling approach that is compatible with agile and all other methodologies. This session also features a case study in which data modeling was introduced part-way through a major initiative that would have failed otherwise, highlighting metrics that illustrate the contrast when utilizing a lean approach and skilled data modelers versus a development-only approach.

Published in: Data & Analytics
  • Be the first to comment

Lean Modeling for Any Methodology

  1. 1. 1© 2019 IDERA, Inc. All rights reserved. LEAN DATA MODELING FOR ANY METHODOLOGY JUNE 18, 2019 Ron Huizenga Senior Product Manager, Enterprise Architecture & Modeling @DataAviator
  2. 2. 2© 2016 IDERA, Inc. All rights reserved. Proprietary and confidential. 2© 2019 IDERA, Inc. All rights reserved. PRE-FLIGHT BRIEFING ▪ A brief history lesson ▪ Methodology contrast ▪ The human factor ▪ Data modeling’s increasing value ▪ Case study • Plan vs. reality • Quality metrics • Data modeling impact ▪ Lean principles • And how to apply them to data ▪ Summary
  3. 3. 3© 2016 IDERA, Inc. All rights reserved. Proprietary and confidential. 3© 2019 IDERA, Inc. All rights reserved. A BRIEF HISTORY LESSON (PART 1) TOTAL QUALITY MANAGEMENT (TQM) 1980’s & 1990’s Industrialization (manufacturing) is the basis for systems development:
  4. 4. 4© 2016 IDERA, Inc. All rights reserved. Proprietary and confidential. 4© 2019 IDERA, Inc. All rights reserved. A BRIEF HISTORY LESSON (PART 2) PREDICTIVE ADAPTIVE
  5. 5. 5© 2016 IDERA, Inc. All rights reserved. Proprietary and confidential. 5© 2019 IDERA, Inc. All rights reserved. METHODOLOGIES AND DEFINITIONS ▪ Waterfall • A linear, sequential approach to the software development life cycle (SDLC) • Used in software engineering and product development. • Emphasizes a logical progression of steps. • Requirements -> Analysis -> Design -> Develop -> Test -> Deploy -> Maintain ▪ Agile • Software development based on iterative development • Requirements and solutions evolve through collaboration • Self-organizing, cross-functional teams • “Increases productivity and reduces time to benefits relative to waterfall” • Variants • SCRUM • Extreme Programming (XP)
  6. 6. 6© 2016 IDERA, Inc. All rights reserved. Proprietary and confidential. 6© 2019 IDERA, Inc. All rights reserved. SCRUM ▪ A lightweight process framework for Agile software development ▪ Fixed duration iterations called Sprints (30 days) ▪ Product backlog ▪ Sprint backlog ▪ Self organizing team • Product Owner • Keeper of the requirements • SCRUM Master • Keeper of the process ▪ Daily SCRUM meetings ▪ Sprint kickoff, Sprint Retrospective
  7. 7. 7© 2016 IDERA, Inc. All rights reserved. Proprietary and confidential. 7© 2019 IDERA, Inc. All rights reserved. EXTREME PROGRAMMING (XP) ▪ The most specific (Radical) of the agile software development frameworks ▪ Five values of XP: • Communication - face to face discussion with white board • Simplicity - “what is the simplest thing that will work?” • Constant Feedback - build – feedback – adjust • Courage - “effective action in the face of fear” • Respect – respectful collaboration in the team ▪ Practices • User stories • Paired programming • Small Releases • Simple Design • Refactoring • Continuous integration • 40 hour work (maximum)
  8. 8. 8© 2016 IDERA, Inc. All rights reserved. Proprietary and confidential. 8© 2019 IDERA, Inc. All rights reserved. WATERFALL VS AGILE Data Modeling
  9. 9. 9© 2016 IDERA, Inc. All rights reserved. Proprietary and confidential. 9© 2019 IDERA, Inc. All rights reserved. AGILE CYCLES
  10. 10. 10© 2016 IDERA, Inc. All rights reserved. Proprietary and confidential. 10© 2019 IDERA, Inc. All rights reserved. AGILE MISINTERPRETED AND MISALIGNED ▪ Short term project perspective vs. longer term organizational benefits ▪ It’s all about producing usable software in every iteration • Often used as an excuse to shortcut or omit other important deliverables • Data architecture/integration • Documentation • Decommissioning of replaced applications/systems • Sound architecture often overlooked because “the business user didn’t tell us that” • Requirements interpreted too literally ▪ Blind focus on software only • “Models are good documentation, but they are immediately obsolete.”
  11. 11. 11© 2016 IDERA, Inc. All rights reserved. Proprietary and confidential. 11© 2019 IDERA, Inc. All rights reserved. THE HUMAN FACTOR ▪ Scrum vs. Extreme ▪ Self-organizing team concept • Often misinterpreted as role-less (extreme) • Any person can perform any role • Can switch from sprint to sprint (iteration) • No specialization • Reality • A formula for disaster in all but the simplest of projects ▪ Often accompanied by attitude of disdain for data modelers • “They just slow us down” • “We don’t need a data model” ▪ Short-sighted management • Long term compromised in favor of short term project goals.
  12. 12. 12© 2016 IDERA, Inc. All rights reserved. Proprietary and confidential. 12© 2019 IDERA, Inc. All rights reserved. DATA ARCHITECT/ MODELER IN AGILE ▪ Enterprise data perspective ▪ Facilitator • Enabler vs. Gatekeeper ▪ Full engagement in sprint planning • Ensure completeness of deliverables • Prioritization of dependencies ▪ Iterative work style • Many simultaneous deliverables ▪ Collaboration • Work with multiple teams simultaneously • Cross-project focus vs.
  13. 13. 13© 2016 IDERA, Inc. All rights reserved. Proprietary and confidential. 13© 2019 IDERA, Inc. All rights reserved. MODEL TYPES & DIAGRAMS ▪ Data Model Separation • Conceptual Models • Logical Models • Physical Models ▪ Specialized Data Models • Dimensional • NoSQL ▪ Data lineage ▪ Business process models • Provide context
  14. 14. 14© 2016 IDERA, Inc. All rights reserved. Proprietary and confidential. 14© 2019 IDERA, Inc. All rights reserved. DATA MODEL CONSTRUCTS ▪ Full Specification • Logical • Physical ▪ Persistence Boundaries • Business Data Objects ▪ Descriptive metadata • Names • Definitions (data dictionary) • Notes ▪ Implementation characteristics • Data types • Keys • Indexes • Views ▪ Business Rules • Relationships (referential constraints) • Value Restrictions (constraints) ▪ Security Classifications + Rules ▪ Governance Metadata • Master Data Management classes • Data Quality classifications • Retention policies
  15. 15. 15© 2016 IDERA, Inc. All rights reserved. Proprietary and confidential. 15© 2019 IDERA, Inc. All rights reserved. GOVERNANCE METADATA
  16. 16. 16© 2016 IDERA, Inc. All rights reserved. Proprietary and confidential. 16© 2019 IDERA, Inc. All rights reserved. CASE STUDY– AS PLANNED ▪ Supply Chain – Commercial Application Suite ▪ 1 Common Database ▪ 4 Parallel Development Streams • By functional area ▪ Planned Duration: 1 year ▪ Planned Cost: $6,000,000 ▪ Agile Methodology (Extreme & Scrum) • Developers responsible for all design/development • 2 week sprints (iterations) ▪ Weekly budgeted direct staffing costs: $ 92,800 • Did not include business SMEs as they were covered separately in corporate budget
  17. 17. 17© 2016 IDERA, Inc. All rights reserved. Proprietary and confidential. 17© 2019 IDERA, Inc. All rights reserved. INITIAL WEEKS ▪ High defect rate ▪ Backlog growing rapidly ▪ By week 16, 50% of effort being spent addressing defects • Direct cost $46,400/week • Without being addressed, project schedule would need to be extended 40 weeks (additional cost of $ 3.7 million) Excitement! Anticipation! Reality:
  18. 18. 18© 2016 IDERA, Inc. All rights reserved. Proprietary and confidential. 18© 2019 IDERA, Inc. All rights reserved. PROBLEM ASSESSMENT ▪ Define • Defect categories ▪ Measure • Discrete vs. weighted impact • Linear vs. cumulative measurement ▪ Analyze • Time series distribution • Defects per object • Defects vs. opportunities ▪ Improve • Remediation strategy ▪ Control • Comparative metrics
  19. 19. 19© 2016 IDERA, Inc. All rights reserved. Proprietary and confidential. 19© 2019 IDERA, Inc. All rights reserved. System Defects & Rework Requirements Database & Persistence User Interface Business Services Incomplete user stories Incorrect business analysis documents Missing foreign key constraints Missing check constraints Missing default values Incorrect data type Missing index Missing audit columns Incorrect table name Incorrect column name Tables not in 3rd Normal Form Incorrect state transition Calculation Error Logic construct error Incorrect looping or branching Services not invoked Messages Navigation flow Values not sorted in dropdowns Subfile/list overflow Controls not working Missing prompts Screens not user friendly Incorrect service invoked Entity Framework mapping error Business Process has changed Incorrect test cases Defective unit tests 3rd party widget integration problems Missing processes Inadequate Subject Matter Expert Knowledge DEFINE: DEFECT CATEGORIES & IMPACT
  20. 20. 20© 2016 IDERA, Inc. All rights reserved. Proprietary and confidential. 20© 2019 IDERA, Inc. All rights reserved. CUMULATIVE DEFECTS
  21. 21. 21© 2016 IDERA, Inc. All rights reserved. Proprietary and confidential. 21© 2019 IDERA, Inc. All rights reserved. WEIGHTED DEFECT CATEGORIES Defect Category Primary Layer Impact Comments Defect Count Cumulative Count Defect % Cumulative Defect % Database & Persistence Data Layer Database and persistence errors can be very problematic, time consuming and expensive to correct. There is always an impact to the persistence mapping in the business services layer which must be corrected. In addition, changes may also ripple to the User Interface layer. 243 243 35.01% 35.01% Business services Business Layer Errors in business services are typically problems in logic, calculations etc. This could cause erronious data. However, the corrections are usually limited to the business layer itself, and do not require structural changes (and hence mapping changes to the data layer). 212 455 30.55% 65.56% User Interface Presentation Layer UI errors are almost always isolated to the presentation layer and generally fairly straigh forward to fix. 197 652 28.39% 93.95% Requirements any Requirments errors could impact any and all layers, depending upon the severity or scope of the error. They can not be quantified in general and must be examined on a case by case basis to determine impact. 42 694 6.05% 100.00% Total 694 694 100.00% 100.00% Severity Points Weighted Score Cumulative Score Score % Cumulative Score % 7 1,701 1,701 62.95% 62.95% 3 636 2,337 23.54% 86.49% 1 197 2,534 7.29% 93.78% 4 168 2,702 6.22% 100.00% 2702 2,702 100.00% 100.00% A x B =
  22. 22. 22© 2016 IDERA, Inc. All rights reserved. Proprietary and confidential. 22© 2019 IDERA, Inc. All rights reserved. CUMULATIVE DEFECT SEVERITY
  23. 23. 23© 2016 IDERA, Inc. All rights reserved. Proprietary and confidential. 23© 2019 IDERA, Inc. All rights reserved. SPECIFIC DATABASE DEFECT POINT VALUES (SEVERITY) No. Defect Type Description Points 1 Duplicate table 10 2 Table not normalized 10 3 Primary Key Incorrect 5 4 Missing Foreign Key (relationship) 5 5 Referential Integrity constraint incorrect 3 6 Missing foreign key index 2 7 Audit Column missing 2 8 Check Constraint Missing 1 9 Default value not specified 1 10 Incorrect table naming 3 11 Column data type incorrect 2 12 Column NULL specification incorrect 1 13 Incorrect column naming 2
  24. 24. 24© 2016 IDERA, Inc. All rights reserved. Proprietary and confidential. 24© 2019 IDERA, Inc. All rights reserved. DATABASE & PERSISTENCE DEFECTS
  25. 25. 25© 2016 IDERA, Inc. All rights reserved. Proprietary and confidential. 25© 2019 IDERA, Inc. All rights reserved. TIME SERIES DISTRIBUTION OF DEFECTS
  26. 26. 26© 2016 IDERA, Inc. All rights reserved. Proprietary and confidential. 26© 2019 IDERA, Inc. All rights reserved. SMOOTHING – CUMULATIVE ANALYSIS
  27. 27. 27© 2016 IDERA, Inc. All rights reserved. Proprietary and confidential. 27© 2019 IDERA, Inc. All rights reserved. REMEDIATION ▪ Apply LEAN principles to: • Increase efficiency (eliminate waste) • Build in quality • Create knowledge • Optimize ▪ Use Senior Data Architect – Cross Team Focus • Introduced in week 21 of project ▪ Process Changes • Model all changes • Generate DDL from modeling tool • 1 developer dedicated to persistence mapping • Works for data architect ▪ Halt functional design/development to reset • Redesign database • Sprints dedicated to problem cleanup ▪ Target: Reduce data defects by at least 75% going forward
  28. 28. 28© 2016 IDERA, Inc. All rights reserved. Proprietary and confidential. 28© 2019 IDERA, Inc. All rights reserved. OBJECTS & DEFECTS/WEEK COMPARISON
  29. 29. 29© 2016 IDERA, Inc. All rights reserved. Proprietary and confidential. 29© 2019 IDERA, Inc. All rights reserved. DEFECTS PER OBJECT COMPARISON
  30. 30. 30© 2016 IDERA, Inc. All rights reserved. Proprietary and confidential. 30© 2019 IDERA, Inc. All rights reserved. COMPARISON – CUMULATIVE OBJECTS VS. DEFECTS
  31. 31. 31© 2016 IDERA, Inc. All rights reserved. Proprietary and confidential. 31© 2019 IDERA, Inc. All rights reserved. COMPARATIVE Measurement Measurement Period (Weeks 1 -20) Control Period (Weeks 21 - 31) Performance Improvement Interval Length (weeks) 20 11 Objects Created 957 1,083 Defects 1,077 38 Defect Opportunities 4,090 4,333 Defect Points 1,696 87 Defect Point Opportunities 8,886 8,991 Average Objects/week 47.85 98.45 205.76% Average Defects/week 53.85 3.45 1558.82% Average Defect Points/week 84.80 7.91 1072.18% Average defects/object 1.13 0.04 3207.37% Average Defect Opportunities/Week 204.50 393.91 Defects/Opportunity 0.263 0.009 3002.60% Defect Points/Opportunity 0.191 0.010 1972.46%
  32. 32. 32© 2016 IDERA, Inc. All rights reserved. Proprietary and confidential. 32© 2019 IDERA, Inc. All rights reserved. THE BOTTOM LINE ▪ On time completion ▪ Avoided $3.7 million overrun ▪ Senior Enterprise Data Architect + Modeling Tools $200K • Duration of project ▪ ROI: ($3.7 million – $200K)/$200K = 1,750% • Had this been done at the beginning of the project, returns would have been even greater
  33. 33. 33© 2016 IDERA, Inc. All rights reserved. Proprietary and confidential. 33© 2019 IDERA, Inc. All rights reserved. WHAT IS LEAN? ▪ Has it’s basis in manufacturing, and has been adapted to knowledge work • Toyota Production System (TPS) ▪ Organizational focus vs. Agile’s software focus ▪ Repeatable process to minimize waste, maximize value ▪ Requires • Quality standards • Collaboration of specialized workers ▪ Kaizen • “kai-” (change) “-zen” (good) • “continuous improvement” or “small incremental improvements” of all areas of a company
  34. 34. 34© 2016 IDERA, Inc. All rights reserved. Proprietary and confidential. 34© 2019 IDERA, Inc. All rights reserved. LEAN PRINCIPLES ▪ Eliminate waste • Eliminate anything that does not add value ▪ Build quality in • Quality is everybody’s job! • Test driven, incremental development with constant feedback • Automate processes prone to human error ▪ Create knowledge • Properly document and retain valuable learning ▪ Deliver fast • Remove blockers • Don’t over-engineer ▪ Respect people • All aspects: communication, handle conflict, onboarding, process improvement • Empowerment ▪ Optimize the whole • Don’t sacrifice quality for speed • Understand capacity and downstream impact of all work • Identify and optimize value streams
  35. 35. 35© 2019 IDERA, Inc. All rights reserved. AGILE VS. LEAN ▪ Agile ▪ Proposed as “a better way of developing software ▪ Bottom-up focus • Short cycle, frequent delivery ▪ Kanban usage • Fixed duration iterations • Limit time of development • Each iteration begins with a fresh board ▪ Focus is delivering software ▪ Lean ▪ Strategic as well as operational • Improve IT’s value to the organization ▪ Top-down, End-to-End Focus (E2E) • “See the whole” ▪ Kanban usage • Continuous flow • Limit work-in-progress • When a task completes, PULL the next in sequence ▪ Focus is delivering real value • (not just software) “Agile is the new Waterfall”
  36. 36. 36© 2016 IDERA, Inc. All rights reserved. Proprietary and confidential. 36© 2019 IDERA, Inc. All rights reserved. START OF ITERATION ▪ Participate fully in iteration planning ▪ Ensure there is a “Named Release” as of completion of previous iteration • Always have a baseline for compare/merge ! ▪ Submodels • Structure by relevant topic/subject area • At story level if necessary to facilitate communication • Roll up to parent level submodels
  37. 37. 37© 2016 IDERA, Inc. All rights reserved. Proprietary and confidential. 37© 2019 IDERA, Inc. All rights reserved. MANAGING ITERATIONS ▪ Always have a baseline for compare/merge ! ▪ Within iteration workflow • Model each change, associating with appropriate task/user story • Generate incremental DDL script(s) and stage to build server • Use a robust script naming convention, particularly if utilizing automated build systems • 1 data modeler may be working with multiple dev teams simultaneously • Some designs will be originated by data modeler • Others may be from developer “sandbox” − Compare/merge and redesign as appropriate − Ensure developer uses the officially sanctioned script • Create “Named Release” at end of iteration • Create delta script by using compare/merge • Based on Named Release from the previous iteration ▪ Use sub-models for audience specific perspective ▪ Maintain the discipline! ▪ Participate fully in iteration planning and retrospectives
  38. 38. 38© 2016 IDERA, Inc. All rights reserved. Proprietary and confidential. 38© 2019 IDERA, Inc. All rights reserved. ER/STUDIO: CHANGE MANAGEMENT CENTER - TRACEABILITY
  39. 39. 39© 2016 IDERA, Inc. All rights reserved. Proprietary and confidential. 39© 2019 IDERA, Inc. All rights reserved. MANAGING COMPLEXITY ▪ Have an overall plan guiding the initiative • Usually requires analysis and some modeling BEFORE development starts ▪ Some areas may be very complex, requiring multiple iterations to design/develop ▪ Use data model design patterns as a starting point ▪ The “wave” approach • Data modelers working on some items 1 or 2 iterations ahead of the development team • Logical / Physical modeling separation facilitates this • Make changes to logical model in advance • Compare/merge appropriate changes to physical at the right time • Enterprise perspective of the data ▪ Fully documented data models!! • Data dictionary definitions • Documented relationships/role names • The physical model IS the implementation • ALL physical constructs
  40. 40. 40© 2016 IDERA, Inc. All rights reserved. Proprietary and confidential. 40© 2019 IDERA, Inc. All rights reserved. ER/STUDIO – COMPARE AND MERGE
  41. 41. 41© 2016 IDERA, Inc. All rights reserved. Proprietary and confidential. 41© 2019 IDERA, Inc. All rights reserved. GENERATE SCRIPT
  42. 42. 42© 2016 IDERA, Inc. All rights reserved. Proprietary and confidential. 42© 2019 IDERA, Inc. All rights reserved. END OF ITERATION WRAP-UP ▪ Create “Named Release” at completion • Serves as baseline for start of next iteration • Serves as baseline for comparison at ANY later point ▪ Create delta DDL script by using compare/merge • Based on Named Release from end of the previous iteration ▪ Create full database DDL script • Can be used to easily create “sandbox” databases quickly ▪ Ensure the model(s) have been published ▪ Participate fully in planning and retrospective meetings • Lessons learned • Celebrate the successes
  43. 43. 43© 2016 IDERA, Inc. All rights reserved. Proprietary and confidential. 43© 2019 IDERA, Inc. All rights reserved. AUTOMATED BUILD SYSTEM CONSIDERATIONS ▪ Require synchronized deliverables ▪ Database (DDL) ▪ Application code ▪ Persistence • Data services • Framework updates
  44. 44. 44© 2016 IDERA, Inc. All rights reserved. Proprietary and confidential. 44© 2019 IDERA, Inc. All rights reserved. POST FLIGHT DE-BRIEF ▪ Systems development is continually evolving and improving • There have been no brand new, groundbreaking ideas • Derived from manufacturing principles and practices proven to deliver business value • Learn and adapt based on the cumulative body of knowledge • And fit to suit organizational culture ▪ DATA has ALWAYS been important. More companies are recognizing that. • Applications come and go • Companies always want to retain the data! • Data models are more important than ever in order to • Manage complexity • Increase quality • Deliver value • Avoid failure. ▪ Lean principles improve systems development • Value focus • Efficiency • Waste reduction • Customer Satisfaction ▪ Approaches utilizing lean are the most successful • Predominantly adaptive • With predictive capabilities incorporated • Best of both worlds
  45. 45. 45© 2016 IDERA, Inc. All rights reserved. Proprietary and confidential. 45© 2019 IDERA, Inc. All rights reserved. THANKS! Any questions? You can find me at: ron.huizenga@idera.com @DataAviator

×