Developing applications that stand the test of time


Published on

April 15, 2003: “Leveraging Notational Systems: Developing applications that stand the test of time”. Presented at the Collaborative Expedition workshop, sponsored by the U.S. General Services Administration as part of the federal Chief Information Officers’ Council.

Published in: Technology, Business
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Developing applications that stand the test of time

  1. 1. Cover Page  Developing Applications that  Stand the Test of Time  Author: Jeffrey G. Long ( Date: April 15, 2003 Forum: Talk presented at the Collaborative Expedition workshop, sponsored by the U.S. General Services Administration as part of the federal Chief Information Officers’ Council.   Contents Pages 1‐31: Slides (but no text) for presentation   License This work is licensed under the Creative Commons Attribution‐NonCommercial 3.0 Unported License. To view a copy of this license, visit‐nc/3.0/ or send a letter to Creative Commons, 444 Castro Street, Suite 900, Mountain View, California, 94041, USA.  Uploaded June 27, 2011 
  2. 2. Leveraging Notational Systems:Developing applications th t stand th t t of tiD l i li ti that t d the test f time Jeff Long April 15, 2003
  3. 3. Proposed Agenda 1: Current software development paradigm and consequences thereof (15 minutes) 2: Proposed alternative paradigm and consequences thereof (15 minutes) 3: How to implement rules as data (15 minutes) i t )4/15/03 Copyright 2003 Jeff Long 2
  4. 4. Fundamental Hypothesis ofNotational EngineeringMany problems in government science the government, science, arts, and engineering exist solely because of the way we currently represent them. These problems present an apparent “complexity barrier” and cannot be resolved with more computing power or more money. Their resolution requires a new abstraction which becomes th b i of a notational revolution b the basis f t ti l l ti and solves a whole class of previously- intractable problems problems.4/15/03 Copyright 2003 Jeff Long 3
  5. 5. A New Notational System OftenRequires a Change of Paradigm A way of looking at a subject An example, pattern, archetype, or y model A set of unconscious assumptions we have about a subject4/15/03 Copyright 2003 Jeff Long 4
  6. 6. Part 1 The CurrentP t 1: Th C tSoftware DevelopmentParadigm
  7. 7. Current Paradigm Computer applications are defined in terms of algorithms and data Algorithms are the rules which are used to t manipulate the d t d t and rules i l t th data; data d l are distinct The model for this is the abacus4/15/03 Copyright 2003 Jeff Long 6
  8. 8.  Algorithms are implemented as software Everything else, such as inventory y g , y quantities or employee names, is implemented as data p Thus most rules (e g work processes, (e.g. processes decision trees) are implemented as software4/15/03 Copyright 2003 Jeff Long 7
  9. 9. Immediate Consequences Every application requires a lot of software, often >1 million lines – No one can really understand or manage this level of complexity – There are numerous bugs in any software system – Special tools and techniques are required to create and/or manage large bodies of code; these add to overall costs4/15/03 Copyright 2003 Jeff Long 8
  10. 10.  There is a lot of “maintenance” of the software – The software has to be changed as bugs are found and eliminated – Th software has to be changed as th The ft h t b h d the rules change – Rules change as users become more sophisticated – Rules change due to external p g pressures as well (e.g. new languages or architectures; government requirements; competition)4/15/03 Copyright 2003 Jeff Long 9
  11. 11.  The changes to software themselves introduce new bugs – Extensive testing is required in order to have any comfort level – Extensive documentation is necessary, but often skipped; maintenance p g pp ; programmers don’t know why system was so designed4/15/03 Copyright 2003 Jeff Long 10
  12. 12. Additional Assumptions Users know what they want and can communicate that to programmers – Programmers who have experience in an industry can understand user requirements – User requirements can be documented in a form that is less complex than the actual working system g y Software can be designed the same way as any other complex technology4/15/03 Copyright 2003 Jeff Long 11
  13. 13. Results to Date Increasing application backlog 2/6 development projects cancelled; additional 3/6 considered failures True importance of very good designers and programmers i d increasingly i l recognized Bugs have greater and greater consequences; viruses don’t help4/15/03 Copyright 2003 Jeff Long 12
  14. 14.  High turnover of Chief Information Officers (average tenure: 18 months) ( g ) Increasing use of packages – This typically forces changes in work processes, or else requires expensive customization (>2x package p ( p g price)) Increasing use of offshore outsourcing (>35% by 2005?)4/15/03 Copyright 2003 Jeff Long 13
  15. 15. Lessons Learned? Subject experts cannot communicate all requirements to programmers – their expertise took many years to acquire – their own understanding will evolve Subject experts must see working prototypes, not paper representations Subject experts must be able to directly and continuously update rules as needed “Corporate” knowledge must be externalized4/15/03 Copyright 2003 Jeff Long 14
  16. 16. Part 2: The Ultra-StructureDevelopment P diD l t Paradigm
  17. 17. Where and How Should Rules beRepresented in a System? Remove 99% of all rules from the software Represent them in a standard If/Then form (multiple Ifs, multiple Thens) Represent them as records of data within a very small set of tables Distinction between rules and data largely disappears!4/15/03 Copyright 2003 Jeff Long 16
  18. 18. Immediate Consequences Software size is reduced by 2+ orders 2 of magnitude – simpler to create, manage, understand, p , g , , test, document, and teach – remaining software has no knowledge of the th world; it provides b i control l i ld id basic t l logic that knows what tables to check in what order, how to resolve conflicts, etc. Software development team is very small and manageable4/15/03 Copyright 2003 Jeff Long 17
  19. 19.  “Corporate” knowledge is externalized and is in a form anyone can see and understand y Subject experts can enter, change, and otherwise manage rules (corporate knowledge) di k l d ) directly, without going to l ih i programmers for assistance Knowledge is actionable not only by subject experts (e.g. as an encyclopedia) but also by the computer, for reasoning, decision support, etc.4/15/03 Copyright 2003 Jeff Long 18
  20. 20.  Programmers do not need to know or understand all rules, just enough to determine the classes of rules and the proper animation procedures Serious prototyping becomes feasible; p communications with users improves Testing & QA can be far more rigorous Documentation can be more complete4/15/03 Copyright 2003 Jeff Long 19
  21. 21. Results to Date First commercial system built with this approach grew from $13M to $100M from 1986-2002, with $1K/month software expense; became largest independent wholesaler in their industry partly due to low operating costs Other prototypes in language, biology, legal, games, and artificial life seem to confirm basic claims and expectations4/15/03 Copyright 2003 Jeff Long 20
  22. 22. Part 3: How to ImplementRulesR l as D t (A P i Data Primer) )
  23. 23. Analysis of Rules Statement of rules and device for executing them can be different; need not be software for both Rules can be reformulated into a canonical form of “If a and b and c... then consider x and y and z” One informal, “molecular” rule (e.g. “three strikes and you’re out” i b ik d ’ ” in baseball) may b b ll) be translated as many “atomic” rules4/15/03 Copyright 2003 Jeff Long 22
  24. 24.  “If” values specify conditions under which each rule is examined; these are called “factors” “Then consider” values specify addit Then consider addit- ional criteria that must be considered before deciding what to do; these are called “considerations”4/15/03 Copyright 2003 Jeff Long 23
  25. 25.  Factors become primary keys in third- normal form normal-form RDBMS tables Alternate keys can be specified if usefulTTens of thousands of rules can be f th d f l b grouped into 10-50 classes based on their th i syntax and semantics t d ti These classes can be managed easily by the tools of a RDBMS4/15/03 Copyright 2003 Jeff Long 24
  26. 26.  Control logic (“animation procedures”) reads relevant rules, including rules , g about selecting rules, and carries out specified actions p Once established, control logic should not need to change; all changes are made by subject experts who directly update the data (rules) of the system4/15/03 Copyright 2003 Jeff Long 25
  27. 27.  Separation of rules into categories defines tables in the RDBMS There is no practical limit on number of rows in a table (i.e. > 1,000 rules is no problem) Referential integrity and field edits ensure that rules maintain integrity Queries, report writers and forms ease the review of rules by authorized users f4/15/03 Copyright 2003 Jeff Long 26
  28. 28.  Design can proceed by iterative prototype; small prototypes can easily evolve to necessary level of complexity Basic design process is to: – define what exists (existential rules) – define relations between these (network & authorization rules) – d fi processes ( t define (protocol & meta-protocol rules) l t t l l ) The application essentially becomes an Expert System using a RDBMS4/15/03 Copyright 2003 Jeff Long 27
  29. 29. The Ruleform Hypothesis Complex system structures are created by not- necessarily complex processes; and these processes are created by the animation of competency rules. Competency rules can be rules grouped into a small number of classes whose form is prescribed by "ruleforms". While the competency rules of a system change over time time, the ruleforms remain constant. A well-designed collection of ruleforms can anticipate all logically possible competency rules that might apply to the system, and constitutes the deep structure of the system.4/15/03 Copyright 2003 Jeff Long 28
  30. 30. The Theory Offers a Different Way to Look at Complex Systems and Processes observables surface structure generates rules middle structure constrainsgroups of rules f l deep structure 4/15/03 Copyright 2003 Jeff Long 29
  31. 31. The CoRE HypothesisWe can create “Competency Rule Engines”, or CoREs, Competency Engines CoREs consisting of <50 ruleforms, that are sufficient to represent all rules found among systems sharing broad family resemblances, e g all corporations resemblances e.g. corporations. Their definitive deep structure will be permanent, unchanging, and robust for all members of the family, whose differences in manifest structures and behaviors will be represented entirely as differences in competency rules. The animation procedures for each engine will be relatively simple compared to current applications, requiring less than 100,000 lines of code in a third generation language.4/15/03 Copyright 2003 Jeff Long 30
  32. 32. References Long, J., and Denning, D., “Ultra-Structure: A design theory for complex systems and processes.” In Communications of the ACM (January 1995) Long, J., “A new notation for representing business and other rules.” In Long, J. (guest editor), Semiotica Special Issue on Notational Engineering, Volume 125-1/3 (1999) Long, J., “How could the notation be the limitation?” In Long, J. (guest editor), Semiotica Special Issue on Notational Engineering, Volume 125-1/3 (1999) Long, J., "Automated Identification of Sensitive Information in Documents Using Ultra-Structure". In Proceedings of the 20th Annual ASEM Conference, American Society for Engineering Management (October 1999)4/15/03 Copyright 2003 Jeff Long 31