Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Value Driven Development by Dave Thomas

1,418 views

Published on

Agile, OOP... are like good hygiene in the kitchen, it results in meals with consistent quality and predictable prep and service times. It doesn't result in great meals nor substantially impact the ROI! Lean Thinking clearly shows that the only way to make a significant impact is to improve the value chain by improving flow. If everyone is following best practices no one has competitive advantage. Major improvements in the value chain depend on continued disruptive innovations. Innovations leverage people and their ideas. We use case studies to illustrate the different business and technical innovations and their impact. We conclude with a discussion of how to build and leverage an innovation culture versus a sprint death march when dealing with high value time to market projects.

More details: https://confengine.com/agile-india-2017/proposal/3608/value-driven-development-maximum-impact-maximum-speed

Published in: Engineering
  • Be the first to comment

Value Driven Development by Dave Thomas

  1. 1. 3/11/17 1 © 2014 Bedarra Research Labs. All rights reserved. Value Driven Development Maximum Value, Maximum Speed – Targeted Innovation Trumps Systemic Transformation Dave Thomas www.davethomas.net Chief Scientist, Kx Systems. Chairman YOW! Conferences, Carleton University, School of Computer Science, Canada Outline • Scaling Agile – Can we? Should We? • Process Driven Development • Value Driven Development • Techniques to reduce time, increase value • Innovations Opportunities • Legacy and New Product Case Studies
  2. 2. 3/11/17 2 Scaling Agile – Can it be Done? Yes! If and only if: 1. Has strong consistent and respected leadership. 2. The business adopts a product culture versus a project culture. 3. Implements Continuous Integration and Delivery. 4. Implements a comprehensive measurement system to enable continuous improvement. 5. Establishes a Technical Ladder and Communities of Practice © 2014 Bedarra Research Labs. All rights reserved. Leadership Matters! The Foundation for Trust and Respect A leader has integrity; stands by values, Leads by example, Executes with transparency, Has the wisdom of experience , Articulates the vision, Says when they are wrong, Judges fairly, Promotes trust, Delegates authority, Coaches versus directs, Promotes constructive diversity, Makes timely consistent decisions, Sense of humour, Positively motivates and Educates, Always tries to find the best wrong answer, Has time to listen… ©2011 Bedarra Research Labs. All rights reserved. “Agile Leadership is Fragile” Nigel Dalton, GM Lonely Planet, now CIO Realestate.com
  3. 3. 3/11/17 3 Scaling - Your Challenges => Potential Solutions 3:45 PM Track 3 Lean and Agile in the Large - An informal Q&A for attendees to bring forward their challenges and learn how these challenges have been dealt with in our experience with Lean and Agile in the Large Transitions. Experiences from Eclipse, Instrumentation, Online Travel, Mobile Telecom, Medical, Business Intelligence, Insurance and Banking, Enterprise Software, Defense and Embedded Systems © 2014 Bedarra Research Labs. All rights reserved. Scaling Agile - Should it even be the goal? Agile is not about increased - revenue; customer interactions; productivity; cost savings; new practices which is where most companies have acute demands. Systemic change requires substantial sustained investment in changing the way people do things. It seldom has a major impact on value or cost which is the place companies most need it. Scaling Process often dilutes training to following prescribed steps rather than encouraging thinking based on measured continuous improvement. © 2014 Bedarra Research Labs. All rights reserved.
  4. 4. 3/11/17 4 Process Driven Development Lean and Agile in the Large works but • 10-15% improvement in productivity, 25 %+ quality, little change in productivity • Seldom has the Impact or Agility expected by executives! • Encourages micro improvements, rather than high impact changes! Sprinting to Technical Debt.. • Biased to tools and technologies of the past! • Systemic change expensive to implement and sustain! You can’t lead using yesterdays process from someone else! Process lags best practice by 3 – 5 years! A generic solution is very seldom as effective as a strong specific solution We have great confidence in your abilities to meet our competitive challenges. Hence we have given you Objects, Agile, Tools, Hardware, New Offices, Mentors because we believe you need these to do your best. Here is what the business needs from you: 1. We need to be much faster in selecting and executing high value projects. 2. We need to turn up the dials on productivity while maintaining quality and features. CEO – All Hands Challenge As long as we keep building software the same way as everyone else were have no competitive advantage. …. THIS ISYOUR MISSION…* THIS MESSAGE WILL SELF DESTRUCT IN 30 SECONDS…. * Mission Impossible
  5. 5. 3/11/17 5 Better, Faster, Cheaper – A New Road? We must adapt and improve our way of building software to meet the challenge. 1.Focus on Value – Target resources and innovations to where they will make a difference. 2.Advance our practices – use alternative techniques to better communicate, design, estimate, build, test and deploy. 3.Refactor our organization to streamline and enable more concurrency and reduce cycle time without reducing quality. We need leverage what works and not be constrained by current best practices. If it is slow it has to go! 4.Explore and Experiment – we need to envision alternatives and evaluate them quickly before betting too much on any approach. We need to fail fast to maximize ROI and time. © © Dave Thomas Bedarra Corp 2016 Value Driven Development 1. Identify a high value bottleneck which if accelerated will make a large impact on the business. 2. Quickly envision a solution and experiment • Think Out of the Box (Innovate) • Use the appropriate HW • Use appropriate SW and practices 3. Validate at scale – this is the major reason clever solutions fail to go into production. 4. Complete the work in 3 – 4 months © 2014 Bedarra Research Labs. All rights reserved.
  6. 6. 3/11/17 6 Management Buy In – ROI and Risk Mitigation Business • Strong Senior Business Sponsor • Clear and tangible measureable goal • ROI model shows significant business value ( >15%) • Implementation Timeline of 3 – 4 months • Minimal Disruptive Impact on Business Operations Technical • Single small team tech plus business with track record • Localized code changes • Proof of Concept validation in weeks • Proof of Scale validation in weeks • Easy to monitor SLA Life cycle costs... • Straight forward deployment • Minimum dependency • Independent Acceptance Testing Some Techniques For Fast Delivery with Cases 1. Envisioning 2. Leverage Innovation 3. Rigorous Testing 4. Simplicity 5. Fast Hardware 6. Ship Less Code – Klocs Kill 7. Table and Data Driven 8. Reduce APIs – Improve Integration 9. Loose Coupling 10. Simple Data Architecture 11. Collection Oriented Programming (just enough FP) © © Dave Thomas Bedarra Corp 2016
  7. 7. 3/11/17 7 Envisioning => Tangible Requirements • easily expressed by Business • easily understood by IT • acceptance easily understood by Business • acceptance easily understood by IT • acceptance criteria => acceptance test (AT) ©2004 Bedarra Research Labs. All rights reserved. Story Competitive Delta Analysis Customer Field Studies & Interviews Technology Evaluations Market & Product Analysis Brainstorming & Visioning QFD House of Quality Prototyping Acceptance Criteria Story Acceptance Criteria Envisioning Practices Envisioning – Requirements Through Design The Waterfall Pitfall Plan ‘everything’ before you do ‘anything’ until… It’s 2 years late, 173% over budget, kind of buggy, & costs $32,345.99… But “it does everything you’d ever want to do!” The ‘Potential’ Agile Pitfall “Planning, shamming, I need to add 1+1, so let’s just build it!” On time, on budget, and we’re giving it away as a beta. “It’s bug-free, it works, and it adds 1+1!” Darn… now how do we add all the features we never had time to think about? Envisioning gives Agile some breathing room… Allows us to understand enough of the vision of ‘tomorrow’…
  8. 8. 3/11/17 8 Low-Fidelity Envisioning Prototype Example Low-fidelity prototype § Initially rough and then later refined drawings § Interactive branching allowed walkthrough § User model, task model, task flows § 3 structure and navigation alternatives § 2 visual form alternatives Concept iterations § 6 iterations (expanding from 8 to 48 screens) § 3 sprints § 3 internal / 2 external customer sessions Detail iterations § 3 iterations (148 screens) § 8 sprints § 3 internal / 1 external customer sessions Investment § Less than 2% of overall effort Leverage Innovations Improved Business Practices • Simplification, Partnering, Regulatory … Improved Hardware • Performance, Capacity, Latency, Functionality – atomic clock, mobility, voice, cloud … Improved Software • Usability, Productivity, Flexibility – Database; Fault Tolerance – isolation, Languages, Robust Libraries Improved Practices • Property Based Testing, Micro Services, Continuous Deployment, Functional Programming, Algorithms, Self Service … ©2009 Bedarra Research Labs. All rights reserved.
  9. 9. 3/11/17 9 Testing is still 30 – 50% of Development Test Soon – TDD, Inspections, Acceptance Tests • It all starts with acceptance criteria (including ilities – there are no nonfunctional requirements!) • SBE/Cucumber Good | Bad News? Side by Side Comparison of Two Full or Partial Implementations Dog Fooding for UI Property Based Testing © 2014 Bedarra Research Labs. All rights reserved. Testing is still 30 – 50% of Development Property Based Testing • QuickCheck is currently the best exemplar • Generates random tests from a spec • Collapses False Positives to minimum fail test sequence • RIAK, Level DB experience Dog Fooding Still Best for UI Testing • Every time a significant amount of code has changes, Freeze the code for 2 – 3 days • Have developers test and log bugs on the code of others. • Fix the bugs for, usually 2 – 3 days. © 2014 Bedarra Research Labs. All rights reserved.
  10. 10. 3/11/17 10 Simplicity! - The Road Not Taken? © 2012 Bedarra Research Labs ComplexitySimplicity Fast Software FAST Software = FAST Hardware + Simplicity Simplicity – Software Sympathy* • Simplicity in Code • Simplicity in Architecture • Functional Thinking * Mechanical Sympathy – Martin Thompson © 2014 Bedarra Research Labs. All rights reserved.
  11. 11. 3/11/17 11 Let the Hardware Do The Work! $15,000 buys a computer 1 TB RAM with 100 TB disk and 64 cores! • Automated Build and Test • All interesting data is in memory! Data Base is an oxymoron . Soon 10TB or persistent memory in a PC! • Flexibility of Data versus Code • Inexpensive Data Conversion/Translation • Data Compression and Encryption is “free” on multi- core • Speed and Memory enable Simpler Algorithms and Architecture • Enable End User Computing e.g. Excel and R versus C#, Java ©2009 Bedarra Research Labs. All rights reserved. Ship Less Code! © 2012 Bedarra Research Labs. All rights reserved. Klocs Kill! • The more code you ship the more code you need test and fix! Abstractions Bloat! • Frameworks inject dependencies into your code! • Object Bloat Costs • Development Time • Execution Time and Space Dependencies Strangle!! • Increase Integration Testing • Increase Build and Deploy Time • Frameworks pull in much than required Object Refactoring harder than Changing Data and Functions! Use Less Objects and Less Code !
  12. 12. 3/11/17 12 Table Driven Programming A picture is 1000 words, a table 400 and a diagram 50 Advantages • Easily understood by Business, BA, Dev and QA • Easy to create, refactor and extend using Excel • Modularity through structured tables • Consistency /Completeness Checking • Easy to version and Diff • Efficient Automated Data Driven implementation • Data Driven means changes can be “hot deployed” to a running application Applications Insurance, Banking, Taxation, Healthcare, ATC, Real-time... ©2009 Bedarra Research Labs. All rights reserved. © 2012 Bedarra Research Labs. All rights reserved. Rules Decision Table Calculation Spreadsheet Data Validation Domain and Range Table Mapping Lookup Table Flow Data,Work Flow, Message Events, Matches State Table Process, Reports Input-Output Table Acceptance Criteria Domain Models Entity-Relationship Attribute Data Dictionary Table Driven Programming Examples
  13. 13. 3/11/17 13 Reduce Integration Time and $$$ Avoid Slow and Complex APIs • Leverage stable physical data formats – pages; logs • Make R/O SQL replicas for fast reporting • Put ATOM/RSS feeds on our legacy/partner systems – journal files, events … • REST and JSONify your services • Use ODBC as a simple interface to complex server systems • Use a simple Mashup tool against to deliver a integrated application view ©2004 Bedarra Research Labs. All rights reserved. Enabling Loose Coupling • All APIs are value based and where possible stateless • Isolation of services in separate processes/machines • Simple Pipes and Filters when possible (data flow) • Occasionally Disconnected – replication and sync; event source.. • Simple efficient implementations using co-routines.. • Orchestration/Composition using Scripting • Service Bus, Messaging.. ZeroMq, Rabbit • Node.js e.g. Azure Mobile Services • Erlang, Actors, Scala Akka ©2003 Bedarra Research Labs. All rights reserved.
  14. 14. 3/11/17 14 Batching, RDB + HDB, Event Source, Lambda Architecture, Gartner HTAP, Forrester Translytical DB © 2013 Bedarra Research Labs. All rights reserved. • Using batching to improve through put • Real-time DB + Historical DB (kdb+) • Real-time (Twitter Storm) + Batch DB • Immutable Data For Space and Speed AVOID Binary Formats 1. Textual data formats such as XML, JSON, CSV are very expensive to parse 2. Textual data formats are bulky (10X) to store 3. Make sure your format has a version id for every format (aka schema change) else working with schema less data is problematic. 4. NOSQL – Easy to Write, Expensive to Store and Process!!! © 2014 Bedarra Research Labs. All rights reserved.
  15. 15. 3/11/17 15 Collection Oriented Programming • Programs written with a functional idioms of map, filter, reduce… are : • More concise and easier to read. • Reduce side effects (leverage immutability) • More amendable to optimization • Data + Functions • Examples • Underscore.js; CoffeeSript, Elm.., ES6 • LINQ, Rx, Reactive Streams, Python .. • Clojure, Scala, Haskell, Erlang, Ocaml, q, Java 8+, C# 4.0 , F#… © 2014 Bedarra Research Labs. All rights reserved. Innovation Opportunities/Insertion Points ©2009 Bedarra Research Labs. All rights reserved. Data and Flow Interfaces • Database Interface • File Interface • Sterilization Interface • Messaging Interface • Functional Transformers - ETL; Map Reduce; GPU … • Query Interface • Disk/San Interface • Shared Memory Interface • Log Interface – Event Sourcing; Historical DB (Batch) + Real-time DB (Stream) • Sync Replicate Interface - CDN leverage; Mobile Occasionally Disconnected; BASE Fault Tolerant • Reactive MVC Interface • Small computational bottle necks • Highly structured rules/calculations • Points of high variability/constant change • Points with large numbers of defects Selective Code Focus
  16. 16. 3/11/17 16 Insurance Legacy + SOA + EAP Challenge Legacy L1, L2, L3,L4; Vendor EAP V1, V2,V3,V4,V5; Integration Services I5, I6; Commercial Insurance V1, L1 Personal Insurance L2; Vehicle Insurance V2 Rating Engine L2, L3, V3; Billing System V4 Policy System of Record L4 + I5 + V5 Enterprise Software Bus I6 Solution 1. Outstanding BAs define all products in tables 2. Agile experts generate applications from BA tables 3. SI experts build simple interfaces to ESB + Interface Acceptance Tests 4. All vendors required to deliver acceptance test HR System Bottleneck Challenge • calculations very complex. • analysts capture in Excel. • 100 Devs in COBOL too slow • Agile OO gave only 15% in productivity Solution • Retain 2 Agile OO experts • Excel rule checker in java • Spreadsheet in Java on mainframe • Legacy team deployed on other systems
  17. 17. 3/11/17 17 Product at Risk – Massive Data Migration Challenge Legacy Database Migration required due to DB Vendor Risk Product Vendor dependent on DB vendor Customers want improved query and reporting Customers have massive data locked in DB vendor at risk Solution 1. Product Team implements ODBC interface to legacy data 2. Product Team migrates product to ODBC and provides improved query and reporting . . . BUT Customers and Product locked to legacy physical data! and Product needs to change schema! 3. Expert SI retained by Product to perform high performance bulk conversion to modern database. (Agile DB Refactoring) 4. Independently developed data comparison program used for acceptance testing © 2010 - 2011 Bedarra Research Labs. All rights reserved. Modernize Legacy Factory Automation Challenge $$$ investment in proprietary automation system Must to integrate new systems and new GUI-based control rooms Solution 1. Modify interface to control units using TCP/IP 2. Implement HTTP and ATOM 3. Client devices are Web 2.0 appliances or PCs
  18. 18. 3/11/17 18 Cyber Analytics – Analyst for Kx 1. Order of magnitude improvement in productivity § Interactive, Incremental, Exploratory with rapid deploy to production § Scripting/Domain Oriented Programming by Specialist Analyst § Analyst/End User Visual Query, Transformation and Visualization at scale § Provide full provenance for all artefacts and analysis § Granular Cross Jurisdictional Collaboration 2. Fast Data § Efficient computation over large volumes of data (at least 10s – 100s TB) § Minimal Impedance between persistent (historical) and live data (real-time) § Enable Analytics from Sensors to Cloud § Interoperable with existing data formats and tools Analyst and Expert Programming Models • 100:1 ratio analysts to experts • Experts surface new functionality to analysts as DSL library extensions Expert Programming Model • Wide Spectrum Functional Vector Language • Fully Interoperable with current technologies – C to OO to Web Analysts Application Programming Model • Wide Spectrum DSLs ( SQL, Spreadsheets...) • Narrow Domain Specific DSL (IP packets, finance, geo, cultural ...) • Leverage existing standards and user models • Interoperable with R, MatLab, MS Office... Core VEE Platform – High Performance Vector Functional Runtime • Data Store Features • Interoperability Features • Core Libraries New Functionalit y
  19. 19. 3/11/17 19 © 2014 Bedarra Research Labs. All rights reserved. Deliver Value Identify the High Business Value Find the simplest innovative solution to delivery quickly Shukriya! `

×