Case study of rules as relational data


Published on

2008: "Case Study of an Enterprise System That Represents Rules Primarily as Relational Data Rather Than via Code". Published in Acta Systemica Vol. 8 No. 2 (2008) pp. 47‐54 available at

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Case study of rules as relational data

  1. 1. Cover Page   Case Study of an Enterprise  System That Represents Rules  Primarily As Relational Data  Rather Than via Code  Author: Jeffrey G. Long ( Date: 2008 Forum:  Acta Systemica Vol. 8 No. 2 (2008) pp. 47‐54 available at   Contents Pages 1‐6: Preprint of paper.   License This work is licensed under the Creative Commons Attribution‐NonCommercial 3.0 Unported License. To view a copy of this license, visit‐nc/3.0/ or send a letter to Creative Commons, 444 Castro Street, Suite 900, Mountain View, California, 94041, USA.  Uploaded June 24, 2011 
  2. 2. Case Study of an Enterprise System That Represents Rules Primarily As Relational Data Rather Than via Code Jeffrey G. Long jefflong@aol.comAbstractBusiness rules are often narrowly defined, but they should include any and all rules that anorganization follows to manage its affairs. These include all work processes, pricing rules,applicable laws and regulations (e.g. taxes), ontologies (e.g. defining the categories by whichcustomers or products may be grouped), taxonomies (e.g. defining how a product or customer isactually categorized), and other kinds of rules. There is a distinct subset of these that are thetechnical interface rules which define various interfaces, such as with users (through forms andreports), data (via database management software), and other systems.In traditional software applications such rules are typically encoded as parameter-drivensoftware. Because there are many rules, applications frequently require millions of lines ofcode. Such a large code corpus ensures significant life-cycle maintenance costs, as onlyprogrammers can update the code, and the code must change as the rules change over time.The approach we’ve used, called “Ultra-Structure,” is to remove any business rules that mightever change from the software, leaving only the control logic for a “competency rule engine” assoftware. All the rest of the rules are represented as data. We use and advocate a relationaldatabase approach to representing rules because of referential integrity, normalization, security,and other benefits offered by any modern RDBMS. Results are presented from a 23-year casestudy of a particular company.Key Words: Enterprise System; Databases; Business Rules; Software Economics; BusinessProcess ReengineeringProblems of Software DevelopmentOne of the most vexing problems in the software industry is why over the past 50 years software(and software development) has not improved exponentially in its effectiveness and efficiency,in a manner comparable to hardware. While even home computer users can now afford aterabyte of data storage, writing code is nearly as laborious and error-prone as it ever was. Inspite of decades of research and development, many software systems either do not or cannotmeet users’ current or future requirements. Their construction and maintenance cost and timeare also extreme.Solving this will not merely require that analysts ask better questions of users, for users can onlysay what they know now: how future government or other regulations, or mergers or divestitures, 1
  3. 3. or new ideas or technologies, may affect their requirements are inherently unknowable.Furthermore, user understanding of their requirements evolves, especially under the guidance ofan analyst who asks about logically-possible alternatives to whatever the user posits. They canalso evolve when an analyst identifies for them new ways that computers or other decision-making technologies might assist them.Solving this will also not merely require that programmers become more efficient, for whatprogrammers encode is the rules of the business, which are subject to constant change. CurrentIntegrated Development Environments and libraries of software classes can only address asnapshot taken at a given time; changing code in response to changing user requirements willalways be required. And when business rules are dispersed over hundreds of thousands of linesof code, even a well-designed system poses maintenance challenges.If important user requirements are inherently unknowable and unpredictable, then how can anysystem ever be designed? Most other areas of engineering have been remarkably successful inpredictably delivering results. But applications such as bridges and homes, while complex, arephysical systems where user requirements tend to change little or within limited areas. Thestandard engineering approach that moves from analysis of requirements, to design, toconstruction of the system, is excellent for these situations. But knowing all likely userrequirements is a luxury not available in software design. Nor should it be, for software capturesthinking in ways that other technologies do not, and thinking will always evolve. Yet we mustaim to design systems that can act “intelligently” and can change greatly over time.This can be done by representing rules in a different and better manner. It requires convertingrules from their natural language form (e.g. as in a policy manual) into one or more rules in acanonical form; categorizing those rules into a small number of formats called “ruleforms” thatare defined by their form and meaning, such that any logically possible rule pertaining to thatapplication area (e.g. order processing) can be expressed somewhere in the system; andmanaging these rules as records in various tables (one per ruleform) through a relational databasemanagement system. Thus, business rules are represented not as software, and not as data inXML tags, but as records (relations) in a modern relational database management system, suchthat referential integrity and security of the rules is guaranteed, and information can readily beaccessed through various tools such as queries and/or report writers.One key benefit of this approach is that subject experts (e.g. trained business managers) can readand manage rules directly, without having to first explain them to programmers. This can helpgreatly as companies try to better manage their knowledge resources by making them external tothe subject expert. Another key benefit is that it facilitates changes to work processes. This canhelp greatly as companies try to better manage their work processes by adapting them to newtechnological, regulatory, competitive, or other constraints.In this approach software is seen as essential to the solution, but is not itself the solution. Insteadsoftware is a problem that must be eliminated whenever possible. 2
  4. 4. Case Study (1985-2008)The Company we are discussing here is a privately-held company which currently has about 50warehouses throughout the United States. They are wholesalers of a commodity product, and arethus competing purely on price and service. Their customers are thousands of retailers who thenresell the product to an end-customer who may be waiting in their office. The Companynormally delivers orders within a few hours of order placement. It processes thousands of ordersa day at its various branches, as well as from the Internet. In this competitive environment it hasgrown to become the largest independent wholesaler (i.e. not a manufacturer) in its industry.Intellinomics Era (1986-2002)Starting in September 1985, my former consulting firm (Intellinomics Corporation) built anenterprise system for the Company. At that time they had about USD $13 million in revenues.This system encoded as data only the interface rules regarding the format of screens and reports,user access authority, data structures, data retrieval, etc., while hard-coding in software thebusiness rules such as work processes. The development cost of the system was about USD$250,000; it ran on a Digital Equipment Corporation VAX computer, had a character-basedinterface, and required about 35,000 lines of C code. It supported nine warehouses and about 50users. The system was installed in September 1986 and was used by the Company for 16 years,until 2002. During this time the Company grew ten times as large, to about USD $130 million.It had no programming staff and used about ten hours per month of contracted programmer time.Industry-Leading System Era (2002-2005)The Company decided to move to a package that was the most widely used among wholesalersin the vertical market of the Company. Few if any requirements were discussed, and the systemwas installed in 2002. The vendor made many assumptions about how wholesalers in thatvertical market would/should operate, but the Company had its own internal work processes andontologies that were different. The system ran on an IBM minicomputer under AIX. It provideda character-based user interface and a highly unusual non-relational database system. After 18months of working with the vendor, the system was deemed unacceptable.CoRE650 Era (2005-present)In March 2003 my current firm, CoRE650 Solutions LLC, started development of a new system.The Company requested the use of off-the-shelf software as much as possible, and as a directresult the system did not initially attempt to encode as data the interface rules, but instead hard-coded them using standard .Net development tools (form painters, etc. of the IDE). However,most other kinds of business rules were encoded as data this time. The resulting system requiredabout 500,000 lines of C# code (excluding blanks and comments) and 160,000 lines of T-SQLcode (including blanks and comments). It runs on a database server and terminal servers usingMS SQL-Server and Windows Server 2003, supporting about 500 users having thin clients in thefield. 3
  5. 5. The system was installed in September 2005. The development cost of the system was aboutUSD $3.4 million, and an additional $1.1 million has since been spent on various fixes andupgrades. During this time the Company grew an additional 40%, to about USD $175 million.The monthly work hours dropped sharply after installation and continue to fall, as shown inAppendix A. There is still a team of five people performing a total of about 300-400 hours ofwork per month; this is expected to continue declining.Since installation the system has processed nearly seven million different “business transact-ions,” defined as (for example) a sale of one or more items, plus delivery, invoicing, cashapplication, and accounting and inventory entries as necessary. A transaction might also refer toa transfer of products between warehouses, purchases of products from manufacturers, orcredit/debit memos and inventory adjustments. The Company’s work processes continue toevolve, for example (a) the high price of fuel necessitated delivery charges under certainconditions, and (b) the Company decided to charge customers a restocking fee for returns mademore than thirty days after purchase.Analysis of ResultsThe project took longer and was more costly than initially expected. An analysis of actual coststo date versus budget shows a 1.95: 1 ratio. According to Forrester Research, however, midsizewholesalers normally spend about 0.6% of gross revenues on software; using this metric, theactual cost was 89% of the industry average. Appendix B shows a chart of these costs. Whilethe budget overrun was of course not good news, the comparatively low ongoing maintenancecost will provide a continuing savings that goes directly to the bottom line of the Company farinto the future.The main drivers of increased project cost were as follows:Converting almost all initial data, including transactions in progress, from a non-relational to arelational database required a great deal of conversion coding; in addition many problems/errorswere found in the source data that had to be resolved before installation. The problem wascompounded because the vendor could not or would not provide documentation for its databaseschema. This problem was unique to this situation; it is not likely to be repeated, although anydata conversion can be difficult. We have no special tools or theories to help with this.To meet the COTS requirement, we used standard Microsoft .Net tools, but found that a greatdeal of code – about five times what was expected -- was required to be written, primarily for theuser interface. We are now moving to a design that still uses .Net but where many of these rulesare represented as data, so that code can be eliminated and future changes to user interfaces willbe far easier to manage. It would have been better to design this flexibility in from day one.When the project started to become overdue, the design process was curtailed and discussionswere minimized “until after cutover.” This resulted in one major subsystem (cash application)being designed and implemented in a more complex manner than was really necessary if the 4
  6. 6. Company’s needs had been better understood. The subsystem then had to be redesigned and re-implemented with correct requirements after further discussions. Continuing to work on userrequirements, when the problem but not its alternatives were known to exist, would have avoidedthis substantial effort.Lastly, by putting together a new team for this project, there was a steep learning curve in threeareas: (a) the team needed to learn how to work together, which was painful, (b) the team neededto learn how to work in a geographically dispersed environment, with various team members allover the United States; and (c) the team needed to learn how to build and test a system where allrules were not easily found in code, but where the programmers and testers had to look in twoplaces – the code and the rulebase – to understand what rules were in effect at any given time. Itwould have been better to have a team that had already worked together on similar projects, butthat was not possible then. There is such a team now.ConclusionsAfter moving most rules to data, the code that is left is mainly control logic that knows nothingabout the world except what tables to look at, in what order, and what to do based on rulesselected for execution. Since this control logic is unlikely to change over time, one benefit ofthis approach is that the software and data structures stay remarkably stable even as the rulescontinue to evolve. Another is that subject experts and business managers can explain new rulesto business analysts (not only programmers), who can then directly update the rules through aRDBMS. In another prior project the subject experts directly updated the rulebase, withouthaving to explain things first to a programmer or even an analyst. This eliminates the possibilityof misunderstandings between subject-area experts and technical experts. Representing rules asdata also means that the RDBMS must be optimized to work efficiently.This project has demonstrated the importance of minimizing the amount of software written andmaintained. This can best be done by implementing both business rules and interface rules asdata, not code. It has also shown the importance of readily changeable work process to facilitatethe continuous evolution of business work processes.ReferencesForrester Research, Inc (2005); US IT Spending Benchmarks for 2005.Long, J., and Denning, D. (1995); Ultra-Structure: A design theory for complex systems andprocesses; Communications of the ACM, Volume 38, Number 1 (pp. 103-120)Long, J. (1999); A new notation for representing business and other rules; Semiotica SpecialIssue: Notational Engineering, Volume 125-1/3 (guest ed. J. Long) (pp. 215-227) 5
  7. 7. Appendix A: Actual Hours Required 2,000.00 1,800.00 1,600.00 Cutover 1,400.00 1,200.00 1,000.00 800.00 600.00 400.00 200.00 0.00 Jun-03 Jun-04 Jun-05 Jun-06 Jun-07 Mar-03 Sep-03 Dec-03 Mar-04 Sep-04 Dec-04 Mar-05 Sep-05 Dec-05 Mar-06 Sep-06 Dec-06 Mar-07 Sep-07 Dec-07Appendix B: Actual versus Ideal Costs and Industry-Standard Costs 200,000 180,000 Actual Industry Standard 160,000 140,000 Cutover 120,000 100,000 80,000 60,000 Ideal 40,000 20,000 0 Jun-03 Jun-04 Jun-05 Jun-06 Jun-07 Mar-03 Sep-03 Dec-03 Mar-04 Sep-04 Dec-04 Mar-05 Sep-05 Dec-05 Mar-06 Sep-06 Dec-06 Mar-07 Sep-07 Dec-07 6