Data are stored in multiple databases and hard to integrate
Modelling is too expensive
Application backlog: don’t ask!
Our Web 2 initiatives are quite separate from core systems
We are moving forward in meeting business needs
Most business functions are provided as services
Many services are orchestrated by our process models and the processes are tracked and constantly improved
All our core static data are integrated and available for analysis
We welcome changes to requirements because they are so easy to implement
or Half full? Half empty?
Why systems projects so often fail
lack of user involvement
no clear statement of requirements
no project ownership
no clear vision and objectives
lack of planning
59% of USA projects are cancelled or overrun budgets Standish Group, 1995, 1997, … 2004,... The need: agility, business focus.
Enterprise Decision Management (EDM) Manifesto
Operational decisions matter
Customers judge you by your decisions: speed, quality, logic, etc.
The effect is cumulative over many small decisions
Many decisions can be automated
This can help with consistency, precision, speed, labour costs, …
… and agility
Mature BRMS and analytics technology exists
Control decisions to get competitive edge
Capture and preserve staff expertise (rules)
Learn from actual sales experience (mine your data)
Give the customer more control (Web 2.0?))
Put the business experts in control (rules, SOA)
Update the rules quickly (rule authoring)
Provide explanation/audit (rules engine)
How to ensure that staff can make small but business critical decisions that
are precisely correct.
meet the customer’s expectations in terms of speed.
are customized to the circumstances of the customer.
are customized to the needs of the business.
meet regulatory requirements.
do not entail big labour cost overheads (more skilled staff, etc.)
How to ensure that the business rules that are used to reach decisions
can be changed easily and quickly.
can be put directly into the hands of the business people.
… separate operational systems from decision support systems
Operational systems manipulate data to control and support business functions
Decision support systems transform and enhance data for the benefit of knowledge workers. These workers must then analyze the results and submit change requests to IT to get any requisite changes actioned.
It is a slow and unpredictable process
Process improvement only takes place at a coarse (or strategic) granularity
The millions of customer-facing micro-decisions embedded in operational systems cannot be upgraded.
Kinds of decision
Decisions that might be candidates for automation:
Subject to regulator’s or management audit
And those that might not:
Low volume and low value
Highly judgmental or intuitive
Socially or politically sensitive
Operational decision automation technologies
Business Intelligence (BI) and performance monitoring, BAM (Business Activity Monitoring)
Data mining and analytics
BI and BAM
Business Intelligence (BI) and
Collects significant aggregate data
Usually not done in real time
Manual feedback to BPM
As BI but includes forecasting and planning
Business Activity Monitoring (BAM )
Real time data collection
Often part of BPMS suites (e.g. Pegarules)
Management dashboards (usually requiring human attention)
A few BAM products can alert apps, services or processes
Extracts aggregate and (sometimes) transactional data from operational systems
Usually not in real time (overnight?)
Requires complex ‘data cleaning’ processes
Provides a single location for data mining and analytics
Good basis for initial EDM projects
Data mining and analytics
Data mining (aka machine learning)
These (and GAs) extract latent relationships, often directly in the form of business rules
Statistics (chiefly regression and clustering)
Optimization (linear and dynamic programming)
E.g. What attributes of lenders make most them likely to default?
What cross-selling strategy works best for over 50s?
What mix of contract options gives the highest margin?
All these techniques can draw conclusions based on actual data (and the assumptions of the models).
BPM – UML activity diagrams Prepare order Assign item to delivery Assign goods to warehouse Pick delivery to satisfy order Assign stock to delivery Order prepared Order satisfied? Y N & priority order Mark order as satisfied Reorder stock Y Reorder needed? Goods arrive New order All orders satisfied Delivery to pick? Y N Goods remaining? Y SALES GOODS INWARD swimlanes
BPM – Use case decomposition with sequence rules enter deal EnterDetails checkLimits commit deal sendConfirms adjustPositions checkDealerLmt checkCptyLmts checkGlobalLmt ConfToCpty ConfToSettlement Sequence diagrams impose design decisions. Therefore... precedes
A BPMN Process
The BPM life cycle Process improvement does not take place in real time
What is service oriented architecture?
A software architectural style that focuses on the use of services to support business requirements.
In an SOA, resources are made available to other participants in the network as independent services that are accessed in a standardized way.
Many definitions of SOA identify the use of web services (using SOAP and WSDL) in its implementation, however it is possible to implement SOA using any service-based technology.
Though built on similar principles, SOA is not the same as Web services . SOA is independent of any specific technologies.
What is service oriented architecture?
Services are loosely coupled; i.e., the service interface is independent of the implementation. There can be unintended consequences of coupling (good and bad ones).
Each SOA service has a quality of service (QoS) associated with it. Example QoS elements are security requirements, such as authentication and authorization, reliable messaging, and policies regarding who can invoke services. SLAs++
Application developers or system integrators can build applications by composing one or more services without knowing the services' underlying implementations.
For example, a service can be implemented either in .NET or J2EE, and the application consuming the service can be on a different platform or language.
Composing loosely coupled services 6 widgets @ 6p 1 6mm toggle @ £1.20 Total cost £1.56 +VAT In stock: All items What do I need to make up a floggle? Bill of materials service Stock control services Pricing service VAT rules service External pricing feed (Honest John’s prices?)
Business drivers for SOA
High project failure rate
Rapidly evolving technology and platforms
The need for agility
High maintenance costs
Rapidly evolving or poorly understood requirements (requirements creep)
The need to innovate new business processes
The need to conform to rapid changes in regulation
The need for greater business focus
The lunatics are running the asylum!
The need for a common language. Involving the business in service definition and delivery
Focus on the real customers for competitive edge
Economies of scale and reuse
Exploiting and integrating the legacy (EAI)
Sharing common services (consistency)
Getting SOA right implementation based services support this interface Service Oriented Architecture support this interface system user real user
BPM and SOA Application & decision services Operational systems/applications Orchestration/choreography layer Business services Business Process Models
Services for the business
SOA (if done properly) –
leads to interfaces about the business, not about supporting the user interface
beware of just wrapping existing interfaces — too low level
must understand the business, not just the computer system
provides services for people to do tasks that deliver them value
They should understand the system in their own terms — not the software developers’ argot
XML, WDSL, UDDI, SOAP, …
WS* – security, reliable messaging, … , etc.
Business Process Modelling
BPEL, BPMN, UML
OWL, SVBR, UML, …
Vertical industry standards
E.g. Insurance (ACORD), Oil & Gas, …
Business rules ‘knowledge packs’
Business rules technology
A business rule is ...
… a compact, atomic , well-formed, declarative statement about an aspect of a business that can be expressed in terms that can be directly related to the business and its collaborators, using simple unambiguous language that is accessible to all interested parties: business owner, business and product analyst, technical architect, customer, and so on. This simple language may include domain-specific jargon. Business rules are always interpreted against a defined domain ontology.
There are several sources of business rules
Use case goals
The architecture of a business rules service Knowledge Base Inference Engine Symbol manipulation, computation, etc. Rules, facts, objects, associations, etc. Chaining, learning, conflict resolution, etc.
Techniques for representing rules
A loan may be approved if the status of the customer is high and the loan is less than 2000 unless the customer has a low rating
Decision trees and decision tables
Equivalent to rulesets
Type models & semantic networks (ontologies)
What are BRMSs?
Using BRMSs together with better requirements engineering and business modelling within the context of SOA should decrease development costs and dramatically shorten development and maintenance cycles.
Business rules management systems separate the rules from data and control logic and maintain them in a repository.
Rules are grouped into rulesets and inference over and within rulesets is both possible and transparent.
BRMSs have applications across all industries and many types of business problem.
In the context of SOA and EDM, the main impact of BRMSs is that process and decision logic can change much more rapidly.
Rules (rulesets) should be regarded as services.
And the business OWNS the rules.
The architecture of a BRMS Rule authoring service Rule engine (decision service) Repository Business services Infrastructural services, including persistence service (not rule-based) Database
Business drivers for BRMS
Current software development practice inhibits rapid delivery; modest changes to existing systems can take too long.
Competitive pressure means that policy and rules must be amenable to rapid change.
Personalizing services, content and interaction styles.
In regulated industries, rules for governance and regulation will change outside the control of the organization.
Sarbanes-Oxley, Basel II, etc. mean that business processes and rules must be visible.
Business rules and processes can be shared by applications across the whole enterprise using multiple channels, thereby encouraging consistent practices.
Benefits of BRMSs
Better alignment and understanding between business and IT
Creation of a common language
Representation of the ‘implemented’ knowledge is more understandable to the business
Faster development (particularly of changes)
Faster maintenance, which particularly relevant in service oriented architectures, where the maintenance of a rules component is addressed outside of the wider IT maintenance context
Clearer auditability of business rule execution
Greater consistency across the enterprise
the same rule executed in different applications
One point of change management
JBoss Rules (DROOLS – BRE only) – free?
Jess – BRE only, Oracle’s rule engine now
PegaRules – BPM focused
Versata – database update focus
Corticon – not rete-based
ILOG JRules (and the .NET product)
Suits Java culture (you have to write your own business friendly language)
Haley Authority (“natural language” authoring)
Suits environments where business people and business analysts may create and maintain some rules
Suits conventional IT culture, business users to not author rules but user management applications can be written
A good BRMS should...
allow business analysts to create and modify the rules
use a fully-featured repository
support backward chaining
allow the rule engine to be a component or service within larger applications
allow applications to be deployed in a service oriented architecture
focus on business rules management (as opposed to just workflow) problems
provide good report generation facilities
provide evidence of successful commercial applications
be compatible with a component-based or service-oriented architecture
offer commercial-standard professional support
Standard solutions to commonly encountered problems
E.g. ‘linked list’, ‘recursive descent’, ‘observer’, ‘ten minute rule’.
Named problems and solutions make it easier to think about and talk about models, designs or processes
Standard solutions make it easier to anticipate details of designs; e.g. when reading code
Easier skills transfer — in digestible chunks
Reduces multiplicity of ways problems are solved; hence:
Easier to discuss and document designs
Easier to read, check, maintain code — fewer surprises
Make your own for your project
job of the chief architect to establish global ways of doing things
project architecture = patterns used on project = ‘how we do things around here’
Originally the notion of a building architect, Christopher Alexander
Actually, Victorian builders
Alexander – built environment
GoF – software design
P4 business process patterns (Aalst, et al ., 2003)
There are other formats, notably for more technical patterns.
Choose a format and stick to it.
Processes as pattern languages ( RulePatterns ) 1. Establish the business objectives 3. Establish the use cases 2. Business process model 7. Timeboxes 9. Automate testing 10. Usability Testing 8. Gradual stiffening 6. User-centred service structure 4. Build a type model (ontology) 5. Discover business rules 13. Ask the business 11. Association loops conceal rules 12. Write the constraints as rules 14. Assign rules to components 16. Policy blackboard 17. Store rules in a repository 18. Encapsulate a reference 19. Determine security model 21. Follow standards 22. Determine ownership & permissions 23. Define a rule- writing style 24. Write the consequent first 15. Base error messages on rules UI Patterns Rule Object Patterns 20. Separate volatile rules Concrete & Terminal Abstract pattern Concrete Pattern Key External Patterns
EDM process and architecture Data warehouse Operational systems External feeds Analytic models Rule authoring software Business expert Customer Decision service Application Rule engine Rule repository
High ROI accretes to systems that...
Have a large number of rules
Have rules that change often
Have rules that involve domain expertise
Have users that are prepared to maintain the rules
Have rules that are complex or interact in complex ways
Store a large amount of data about customers, business processes, etc.
Have clearly defined business processes
Require audit (regulatory controls)
Change management. Securing management ‘buy in’.
Manageable but visible project(s)
Define the long-term architecture
Define the (local) domain model
Start with BRMS introduction and service and decision definition
Add in analytics and optimization
Manage expectations (EDM is not just for Christmas)
Be agile (learn how to descope rather than run late)
Publish the method and architecture (Consider patterns for this)
Education and training (how to talk English!)
Consider fully adaptive control
Running workshops and requirements engineering
Modelling, modelling, modelling. (BPM, use cases and types, ontology, business rules, maths & stats)
Business R ules M anagement S ystems
Agile process management
SOA services portfolio governance
References and further reading
James Taylor and Neil Raden (2007) Smart (Enough) Systems: How to Deliver Competitive Advantage by Automating the Decisions Hidden in Your Business , Addison-Wesley
Project manager’s guide to these ideas
Ian Graham (2006) Business Rules Management and Service Oriented Architecture , Wiley
More technical introduction to the business rules approach
Thomas Davenport and Jeanne Harris (2007) Competing on Analytics , Harvard Business School Press
Management level introduction to predictive analytics