February 5, 2003: “Towards a New Paradigm to Resolve the Software Crisis”. Talk given at University of North Carolina, Chapel Hill.

1. Cover Page
Towards a New
Paradigm to Resolve
the Software Crisis
Author: Jeffrey G. Long (jefflong@aol.com
Date: February 5, 20033
Forum: Talk presented at the University of North Carolina, Chapel Hill.
Contents
Page 1: Proposal
Pages 2‐28: Slides (but no text) for presentation
License
This work is licensed under the Creative Commons Attribution‐NonCommercial
3.0 Unported License. To view a copy of this license, visit
http://creativecommons.org/licenses/by‐nc/3.0/ or send a letter to Creative
Commons, 444 Castro Street, Suite 900, Mountain View, California, 94041, USA.
Uploaded June 27, 2011

2. Towards A New Paradigm to Resolve the Software Crisis
Various kinds of metrics have been used to indicate the scope and magnitude of the
software crisis. Some of these pertain to the high percentage of uninstalled systems;
others to the increasing length of the applications backlog in most shops; others to the
high lifetime cost of developing and maintaining software, whether custom-made or
packaged; and others to the opportunity costs.
All of these metrics indicate that the software business continues to be in a state of crisis
which persists in spite of technical advances such as database management systems;
structured code; improved and more formal development practices; the use of packaged
software, code generators, computer-aided software engineering tools; component-
based, object-oriented, multi-tier software architectures; and colossal budgets. The
problem becomes worse as society expects more from computers and becomes more
dependent upon them. User and organizational frustrations with this problem have
caused CIOs to be replaced regularly, and, increasingly, have resulted in a strong trend
to simply outsource all MIS computer and software operations -- and blame.
This talk presents a new paradigm that has been developed and tested since 1985. It is
based on an analysis of the nature of rules, their underlying common structure, and how
rules are and ought to be represented. The approach is called “Ultra-Structure”.
Rules are typically hard-coded into software applications, and the maintenance of these
rules as they change is the primary cause of subsequent software changes. One
unfortunate consequence of this standard approach is that subject experts must
somehow communicate the rules they wish to see automated to programmers who
typically are not experts in the subject matter of the application; much is lost in the
translation. Another unfortunate consequence is that the software becomes large, often
millions of lines of code, such that no one involved in the project can really comprehend
or manage it. There have been of course numerous initiatives directed at solving these
problems, but the solutions have been ineffectual and the problems they address are
actually secondary and symptomatic rather than primary.
The basic premise of this new approach is that we can resolve all of the major difficulties
with software development, and generate significant new kinds of benefits for users, by
removing most rules and all knowledge of the world from software and instead
representing them the same way we represent data, i.e. as tables in a relational
database.
This approach combines key features of the normally disparate areas of management
information systems, expert systems, and simulations, borrowing the strengths of each
and largely eliminating the known problems of each. The approach can be applied to
any kind of rule-based system, whether the application is intended to support business,
scientific, or other application areas. It has been applied thus far to business and natural
language applications, and an early prototype of its applications to biology has been
developed by Dr. Gidding’s laboratory.

3. Towards A New
Paradigm to Resolve the
Software Crisis
S f Ci i
Jeff Long
Feb 5, 2003
jefflong@aol.com

4. P
Proposed A
d Agenda
d
Current paradigm and consequences
C t di d
thereof (15 minutes)
Proposed alternative paradigm and
consequences thereof (15 minutes)
Technical analysis of rules and their
implementation ( minutes)
p (15 )
General Q&A (15 minutes)
2/5/03 Copyright 2003 Jeff Long 2

5. P di
Paradigm
A way of looking at a subject
f l ki t bj t
An example, pattern, archetype, or
model
A set of unconscious assumptions we
have about a subject
2/5/03 Copyright 2003 Jeff Long 3

6. C
Current S ft
t Software P di
Paradigm
All applications are d fi d i t
li ti defined in terms of
f
algorithms and data
Algorithms are the rules which are used
to manipulate data
The model for this is the abacus
2/5/03 Copyright 2003 Jeff Long 4

7. Algorithms are implemented as software
Everything else, such as inventory
quantities or employee names is
names,
implemented as data
Thus most rules (e.g. work processes,
decision trees) are implemented as
software
2/5/03 Copyright 2003 Jeff Long 5

8. I
Immediate C
di t Consequences
Every application requires a l t of
E li ti i lot f
software, often >1 million lines
No one can really understand or manage
this level of complexity
There are numerous bugs in any software
Th b i ft
system
The ft
Th software has to be changed as b
h t b h d bugs
are found and eliminated
2/5/03 Copyright 2003 Jeff Long 6

9. There is a l t of “maintenance” of the
Th i lot f “ i t ” f th
software
The software has to be changed as the
rules change
Rules h
R l change as users b become more
sophisticated
Rules h
R l change d t external pressures as
due to t l
well
2/5/03 Copyright 2003 Jeff Long 7

10. The h
Th changes t software th
to ft themselves
l
introduce new bugs
Extensive testing is required in order to
have any comfort level
Extensive documentation i necessary, b t
E t i d t ti is but
often skipped; maintenance programmers
don t
don’t know why system was so designed
2/5/03 Copyright 2003 Jeff Long 8

11. Additi
Additional A
l Assumptions
ti
Users know what they want and can
communicate that to programmers
Programmers who have experience in an
industry can understand user requirements
User requirements can be documented in a
form that is less complex than the actual
working system
Software can be designed the same
way as any other complex technology
2/5/03 Copyright 2003 Jeff Long 9

12. R
Results t D t
lt to Date
Increasing application backlog (maybe
use open source software)
2/6 development projects cancelled;
additional 3/6 considered failures
True value of very good designers and
programmers increasingly recognized
Bugs have greater and greater
consequences; viruses don’t help
2/5/03 Copyright 2003 Jeff Long 10

13. High turnover of Chi f I f
Hi h t f Chief Information
ti
Officers (average tenure: 18 months)
Increasing use of packages
This typically forces changes in work
processes, or else requires expensive
customization (>2x package price)
Increasing use of outsourcing upon
surrender (35-45% by 2005)
2/5/03 Copyright 2003 Jeff Long 11

14. L
Lessons L d?
Learned?
Subject experts cannot communicate all
requirements to programmers
their expertise took many y
p y years to acquire
q
their own understanding will evolve
Subject experts must see working prototypes,
not paper representations
t t ti
Subject experts must be able to directly and
continuously update rules as needed
“Corporate” knowledge must be externalized
2/5/03 Copyright 2003 Jeff Long 12

15. A Alt
An Alternative P di
ti Paradigm
Remove most rules from software
Represent rules in canonical form, as
data in a small set of tables
f
Externalize “corporate” knowledge in
relational k
l i l knowledgebase
l d b
Let subject experts manage data
2/5/03 Copyright 2003 Jeff Long 13

16. I
Immediate C
di t Consequences
Software size is reduced by 2+ orders of
magnitude
simpler to create, manage, understand, test,
p , g , , ,
document, and teach
remaining software has no knowledge of the
world; it provides basic control logic that knows
what tables to check in what order, how to resolve
conflicts, etc.
Software development team is very small and
S ft d l tt i ll d
manageable
2/5/03 Copyright 2003 Jeff Long 14

17. “Corporate” knowledge is externalized and is
Corporate
in a form anyone can see and understand
Knowledge is actionable by the computer:
reasoning, error checking, d i i support
i h ki decision
Subject experts can enter, change, and
otherwise manage rules directly without
directly,
programmer assistance
Data no longer merely defines certain details of
the overall system; it defines the identity of the
system
2/5/03 Copyright 2003 Jeff Long 15

18. Programmers do not need to know or
understand all rules, just enough to
determine the classes of rules and the
proper animation procedures
Serious prototyping becomes feasible;
communications with users improves
Testing & QA can be far more rigorous
Documentation can be more complete
2/5/03 Copyright 2003 Jeff Long 16

19. R
Results t D t
lt to Date
Commercial system: 7x growth with
$1K/month software expense
Other prototypes in language, biology, legal,
p yp g g , gy, g ,
games, and artificial life seem to confirm
basic claims and expectations
Closer to SEI Vision: “Th right software,
Cl Vi i “The i h f
delivered defect free, on time and on cost,
every time ”
time.
2/5/03 Copyright 2003 Jeff Long 17

20. A l i of R l
Analysis f Rules
Statement of rules and device for
St t t f l dd i f
executing them can be different; need
not be software for both
tb ft f b th
Rules can be reformulated into a
canonical form of “If a and b and c...
then consider x and y and z”
2/5/03 Copyright 2003 Jeff Long 18

21. “If” values specify conditions under
l if diti d
which each rule is examined; these are
called “f t ” in Ult St t
ll d “factors” i Ultra-Structure Theory
Th
(UST)
“Then consider” values specify
additional criteria that must be
considered; these are called
“considerations” in UST
2/5/03 Copyright 2003 Jeff Long 19

22. Factors become primary keys in thi d
F t b i k i third-
normal form RDBMS
Alternate keys can be specified if useful
Control logic (
g (“animation p procedures”))
reads relevant rules, including rules
about selecting rules, and carries out
g ,
specified actions
2/5/03 Copyright 2003 Jeff Long 20

23. One i f
O informal, “molecular” rule (
l “ l l ” l (e.g. ththree
strikes and you’re out in baseball) may be
translated as many atomic rules
Basic process is to
define what exists (existential rules)
define relations between these (network &
authorization rules)
define processes (protocol & meta-protocol rules)
2/5/03 Copyright 2003 Jeff Long 21

24. Tens of thousands of rules can be
grouped into a small number of classes
based on their syntax and semantics
These classes can be managed easily
by the tools of a RDBMS
Design can proceed by iterative
prototype; small prototypes can easily
evolve to necessary level of complexity
2/5/03 Copyright 2003 Jeff Long 22

25. I l t ti
Implementation
Separation of rules i t classes d fi
S ti f l into l defines
tables in the RDBMS; there is no practical
limit on number of rows in a table
Referential integrity and field edits ensure
that rules maintain integrity
Queries, report writers and forms make
access to rules easy for authorized users
y
2/5/03 Copyright 2003 Jeff Long 23

26. The R l f
Th Ruleform H
Hypothesis
th i
Complex system structures are created by not-
necessarily complex processes; and these
p
processes are created by the animation of
y
competency rules. Competency rules can be
grouped into a small number of classes whose
form is prescribed by "ruleforms". While the
p y
competency rules of a system change over time,
the ruleforms remain constant. A well-designed
collection of ruleforms can anticipate all logically
p g y
possible competency rules that might apply to the
system, and constitutes the deep structure of the
system.
y
2/5/03 Copyright 2003 Jeff Long 24

27. The C RE Hypothesis
Th CoRE H th i
We can create “Competency Rule Engines”, or CoREs,
consisting of <50 ruleforms, that are sufficient to
represent all rules found among systems sharing
p g y g
broad family resemblances, e.g. all corporations.
Their definitive deep structure will be permanent,
unchanging, and robust for all members of the family,
g g, y,
whose differences in manifest structures and
behaviors will be represented entirely as differences
in competency rules. The animation p
p y procedures for
each engine will be relatively simple compared to
current applications, requiring less than 100,000 lines
of code in a third generation language.
g g g
2/5/03 Copyright 2003 Jeff Long 25

28. Ultra-Structure is an Example of
Notational Engineering
Many problems i science, government, th
M bl in i t the
arts, and engineering are caused by the way
we represent them
These problems cannot be resolved with
more computing power or more money
They require a new abstraction which can be
the basis of a notational revolution
2/5/03 Copyright 2003 Jeff Long 26

29. R f
References
Long, J., Denning, D., Ultra-Structure:
Long J and Denning D “Ultra-Structure: A design theory for
complex systems and processes.” In Communications of the
ACM (January 1995)
Long, J., “A new notation for representing business and other
g, , p g
rules.” In Long, J. (guest editor), Semiotica Special Issue on
Notational Engineering, Volume 125-1/3 (1999)
Long, J., “How could the notation be the limitation?” In Long, J.
(guest editor), S i ti S
( t dit ) Semiotica Special Ii l Issue on N t ti
Notational
l
Engineering, Volume 125-1/3 (1999)
Long, J., "Automated Identification of Sensitive Information in
Documents Using Ultra-Structure". In Proceedings of the 20th
Ultra-Structure
Annual ASEM Conference, American Society for Engineering
Management (October 1999)
2/5/03 Copyright 2003 Jeff Long 27